Bug #15694
closed[CWL] workflow hangs after getting 502 Bad Gateway
Description
The workflow status polling thread got "502 Bad Gateway", the workflow runner stopped monitoring/submitting jobs but didn't fail either, it hung:
2019-10-07T09:13:21.337649988Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:21.337649988Z arvados.cwl-runner ERROR: Error checking states on API server: %s 2019-10-07T09:13:21.337649988Z Traceback (most recent call last): 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 369, in poll_states 2019-10-07T09:13:21.337649988Z proc_states = table.list(filters=[["uuid", "in", page]]).execute(num_retries=self.num_retries) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper 2019-10-07T09:13:21.337649988Z return wrapped(*args, **kwargs) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/http.py", line 835, in execute 2019-10-07T09:13:21.337649988Z method=str(self.method), body=self.body, headers=self.headers) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/http.py", line 157, in _retry_request 2019-10-07T09:13:21.337649988Z resp.status if resp else exception) 2019-10-07T09:13:21.337649988Z File "/usr/lib/python2.7/logging/__init__.py", line 1179, in warning 2019-10-07T09:13:21.337649988Z self._log(WARNING, msg, args, **kwargs) 2019-10-07T09:13:21.337649988Z File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log 2019-10-07T09:13:21.337649988Z self.handle(record) 2019-10-07T09:13:21.337649988Z File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle 2019-10-07T09:13:21.337649988Z self.callHandlers(record) 2019-10-07T09:13:21.337649988Z File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers 2019-10-07T09:13:21.337649988Z hdlr.handle(record) 2019-10-07T09:13:21.337649988Z File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle 2019-10-07T09:13:21.337649988Z self.emit(record) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit 2019-10-07T09:13:21.337649988Z "%s: %s" % (record.name, record.getMessage()) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update 2019-10-07T09:13:21.337649988Z current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger) 2019-10-07T09:13:21.337649988Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container 2019-10-07T09:13:21.337649988Z raise e 2019-10-07T09:13:21.337649988Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:29.060662282Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:29.060662282Z arvados.cwl-runner ERROR: Fatal error in state polling thread. 2019-10-07T09:13:29.060662282Z Traceback (most recent call last): 2019-10-07T09:13:29.060662282Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 371, in poll_states 2019-10-07T09:13:29.060662282Z logger.exception("Error checking states on API server: %s") 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 1200, in exception 2019-10-07T09:13:29.060662282Z self.error(msg, *args, **kwargs) 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error 2019-10-07T09:13:29.060662282Z self._log(ERROR, msg, args, **kwargs) 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log 2019-10-07T09:13:29.060662282Z self.handle(record) 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle 2019-10-07T09:13:29.060662282Z self.callHandlers(record) 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers 2019-10-07T09:13:29.060662282Z hdlr.handle(record) 2019-10-07T09:13:29.060662282Z File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle 2019-10-07T09:13:29.060662282Z self.emit(record) 2019-10-07T09:13:29.060662282Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit 2019-10-07T09:13:29.060662282Z "%s: %s" % (record.name, record.getMessage()) 2019-10-07T09:13:29.060662282Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update 2019-10-07T09:13:29.060662282Z current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger) 2019-10-07T09:13:29.060662282Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container 2019-10-07T09:13:29.060662282Z raise e 2019-10-07T09:13:29.060662282Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:43.365414753Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:43.365414753Z Exception in thread Thread-22: 2019-10-07T09:13:43.365414753Z Traceback (most recent call last): 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner 2019-10-07T09:13:43.365414753Z self.run() 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/threading.py", line 754, in run 2019-10-07T09:13:43.365414753Z self.__target(*self.__args, **self.__kwargs) 2019-10-07T09:13:43.365414753Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 386, in poll_states 2019-10-07T09:13:43.365414753Z logger.exception("Fatal error in state polling thread.") 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 1200, in exception 2019-10-07T09:13:43.365414753Z self.error(msg, *args, **kwargs) 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error 2019-10-07T09:13:43.365414753Z self._log(ERROR, msg, args, **kwargs) 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log 2019-10-07T09:13:43.365414753Z self.handle(record) 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle 2019-10-07T09:13:43.365414753Z self.callHandlers(record) 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers 2019-10-07T09:13:43.365414753Z hdlr.handle(record) 2019-10-07T09:13:43.365414753Z File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle 2019-10-07T09:13:43.365414753Z self.emit(record) 2019-10-07T09:13:43.365414753Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit 2019-10-07T09:13:43.365414753Z "%s: %s" % (record.name, record.getMessage()) 2019-10-07T09:13:43.365414753Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update 2019-10-07T09:13:43.365414753Z current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger) 2019-10-07T09:13:43.365414753Z File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container 2019-10-07T09:13:43.365414753Z raise e 2019-10-07T09:13:43.365414753Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway"> 2019-10-07T09:13:43.365414753Z
Files
Updated by Nico César about 5 years ago
- File nginx_error.log nginx_error.log added
Looks like postgres went down.
App 125491 output: WARNING: terminating connection because of crash of another server process App 125491 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125491 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125491 output: [ 2019-10-07 09:13:09.4018 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::ConnectionBad: PQconsumeInput() SSL SYSCALL error: EOF detected App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)): App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first' App 125491 output: from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each' App 125491 output: from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send' App 125491 output: from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads' App 125491 output: from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception' [ W 2019-10-07 09:13:09.4531 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428041] Sending 502 response: application did not send a complete response App 125611 output: WARNING: terminating connection because of crash of another server process App 125611 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125611 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 4145 output: WARNING: terminating connection because of crash of another server process App 125591 output: WARNING: terminating connection because of crash of another server process App 4145 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125591 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 4145 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125591 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125531 output: WARNING: terminating connection because of crash of another server process App 125531 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125531 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125571 output: WARNING: terminating connection because of crash of another server process App 125571 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125571 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125551 output: WARNING: terminating connection because of crash of another server process App 125551 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125551 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 61001 output: WARNING: terminating connection because of crash of another server process App 61001 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 61001 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 114383 output: WARNING: terminating connection because of crash of another server process App 114383 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 114383 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. App 125511 output: WARNING: terminating connection because of crash of another server process App 125511 output: DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. App 125511 output: HINT: In a moment you should be able to reconnect to the database and repeat your command. [ W 2019-10-07 09:13:09.5670 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428042] Sending 502 response: application did not send a complete response App 125491 output: [ 2019-10-07 09:13:09.5668 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server App 125491 output: : SELECT "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)):
[41281413.790132] keep-balance invoked oom-killer: gfp_mask=0x24280ca, order=0, oom_score_adj=0 [41281413.790136] keep-balance cpuset=/ mems_allowed=0 [41281413.790142] CPU: 6 PID: 120486 Comm: keep-balance Not tainted 4.4.0-104-generic #127-Ubuntu [41281413.790144] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017 [41281413.790146] 0000000000000286 a01a7fabf763ace9 ffff8802fc497af8 ffffffff813fb523 [41281413.790149] ffff8802fc497cb0 ffff880249674600 ffff8802fc497b68 ffffffff8120cb7e [41281413.790152] 0000000000000015 0000000000000000 ffff8807ba714300 ffff8807b9e64600 [41281413.790154] Call Trace: [41281413.790162] [<ffffffff813fb523>] dump_stack+0x63/0x90 [41281413.790166] [<ffffffff8120cb7e>] dump_header+0x5a/0x1c5 [41281413.790171] [<ffffffff81393b54>] ? apparmor_capable+0xc4/0x1b0 [41281413.790175] [<ffffffff81193562>] oom_kill_process+0x202/0x3c0 [41281413.790177] [<ffffffff81193989>] out_of_memory+0x219/0x460 [41281413.790181] [<ffffffff81199995>] __alloc_pages_slowpath.constprop.88+0x965/0xb00 [41281413.790184] [<ffffffff81199db6>] __alloc_pages_nodemask+0x286/0x2a0 [41281413.790187] [<ffffffff811e50bd>] alloc_pages_vma+0xad/0x250 [41281413.790190] [<ffffffff811c2c5e>] handle_mm_fault+0x110e/0x1820 [41281413.790195] [<ffffffff8106b577>] __do_page_fault+0x197/0x400 [41281413.790197] [<ffffffff8106b802>] do_page_fault+0x22/0x30 [41281413.790201] [<ffffffff818469f8>] page_fault+0x28/0x30 [41281413.790202] Mem-Info: [41281413.790207] active_anon:5860563 inactive_anon:1043589 isolated_anon:0 active_file:611 inactive_file:643 isolated_file:2 unevictable:8560 dirty:0 writeback:0 unstable:0 slab_reclaimable:37088 slab_unreclaimable:36027 mapped:1582583 shmem:1631218 pagetables:116589 bounce:0 free:48997 free_pcp:30 free_cma:0 [41281413.790211] Node 0 DMA free:15876kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [41281413.790217] lowmem_reserve[]: 0 417 28110 28110 28110 [41281413.790221] Node 0 DMA32 free:111672kB min:960kB low:1200kB high:1440kB active_anon:298292kB inactive_anon:9296kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:507840kB managed:427136kB mlocked:0kB dirty:0kB writeback:0kB mapped:19392kB shmem:19460kB slab_reclaimable:1260kB slab_unreclaimable:1008kB kernel_stack:176kB pagetables:2604kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? yes [41281413.790227] lowmem_reserve[]: 0 0 27693 27693 27693 [41281413.790230] Node 0 Normal free:68440kB min:66580kB low:83224kB high:99868kB active_anon:23143960kB inactive_anon:4165060kB active_file:2452kB inactive_file:2576kB unevictable:34240kB isolated(anon):0kB isolated(file):8kB present:28835840kB managed:28358404kB mlocked:34240kB dirty:0kB writeback:0kB mapped:6310940kB shmem:6505412kB slab_reclaimable:147092kB slab_unreclaimable:143100kB kernel_stack:22672kB pagetables:463752kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:34304 all_unreclaimable? yes [41281413.790236] lowmem_reserve[]: 0 0 0 0 0 [41281413.790239] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15876kB [41281413.790250] Node 0 DMA32: 238*4kB (UME) 1168*8kB (UME) 1373*16kB (UME) 595*32kB (UME) 163*64kB (UME) 53*128kB (UME) 37*256kB (UME) 16*512kB (UME) 13*1024kB (UME) 2*2048kB (UM) 2*4096kB (M) = 111784kB [41281413.790278] Node 0 Normal: 728*4kB (UME) 1259*8kB (UME) 1072*16kB (UMEH) 788*32kB (UMEH) 193*64kB (UMEH) 4*128kB (H) 2*256kB (MH) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 69240kB [41281413.790290] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [41281413.790292] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [41281413.790293] 1633529 total pagecache pages [41281413.790294] 0 pages in swap cache [41281413.790296] Swap cache stats: add 0, delete 0, find 0/0 [41281413.790297] Free swap = 0kB [41281413.790298] Total swap = 0kB [41281413.790299] 7339918 pages RAM [41281413.790300] 0 pages HighMem/MovableOnly [41281413.790301] 139556 pages reserved [41281413.790302] 0 pages cma reserved [41281413.790303] 0 pages hwpoisoned [41281413.790304] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [41281413.790311] [ 446] 0 446 3051 1081 11 3 0 0 haveged [41281413.790313] [ 850] 0 850 4030 213 12 3 0 0 dhclient [41281413.790315] [ 958] 0 958 7548 503 20 4 0 0 systemd-logind [41281413.790318] [ 972] 0 972 17152 523 37 3 0 0 syslog-ng [41281413.790320] [ 974] 0 974 68644 730 37 3 0 0 accounts-daemon [41281413.790322] [ 981] 0 981 7470 56 19 3 0 0 cgmanager [41281413.790324] [ 989] 0 989 6511 52 18 3 0 0 atd [41281413.790327] [ 994] 0 994 1099 23 8 3 0 0 runsvdir [41281413.790329] [ 1000] 0 1000 6932 73 17 3 0 0 cron [41281413.790331] [ 1018] 0 1018 1100 23 8 3 0 0 acpid [41281413.790334] [ 1025] 102 1025 10756 155 25 3 0 -900 dbus-daemon [41281413.790336] [ 1051] 0 1051 1061 21 7 3 0 0 runsv [41281413.790338] [ 1052] 0 1052 1061 21 7 3 0 0 runsv [41281413.790340] [ 1053] 0 1053 1061 21 7 3 0 0 runsv [41281413.790342] [ 1054] 0 1054 1061 21 7 3 0 0 runsv [41281413.790344] [ 1067] 0 1067 1097 30 7 3 0 0 svlogd [41281413.790346] [ 1068] 0 1068 1097 30 7 3 0 0 svlogd [41281413.790348] [ 1089] 0 1089 1097 30 8 3 0 0 svlogd [41281413.790350] [ 1122] 0 1122 1097 31 8 3 0 0 svlogd [41281413.790352] [ 1160] 0 1160 4868 61 16 3 0 0 irqbalance [41281413.790355] [ 1236] 999 1236 20945 4125 35 6 0 0 consul [41281413.790357] [ 1426] 0 1426 3764 47 14 3 0 0 xinetd [41281413.790359] [ 1444] 0 1444 3665 34 13 3 0 0 agetty [41281413.790361] [ 1521] 0 1521 3619 36 12 3 0 0 agetty [41281413.790364] [63492] 992 63492 11319 205 27 3 0 0 systemd [41281413.790366] [63494] 992 63494 41031 680 44 3 0 0 (sd-pam) [41281413.790368] [63673] 992 63673 6538 197 17 3 0 0 screen [41281413.790371] [63674] 992 63674 4988 110 14 3 0 0 bash [41281413.790373] [63703] 0 63703 18569 195 41 3 0 0 sudo [41281413.790375] [63806] 0 63806 18366 155 40 3 0 0 su [41281413.790378] [63807] 0 63807 9676 4849 23 3 0 0 bash [41281413.790380] [48666] 1001 48666 11319 206 25 3 0 0 systemd [41281413.790382] [48669] 1001 48669 41031 680 44 3 0 0 (sd-pam) [41281413.790384] [130006] 0 130006 170933 1670 258 4 0 0 systemd-journal [41281413.790387] [29786] 4047 29786 17384 2854 37 5 0 0 vault [41281413.790389] [96914] 0 96914 377988 96637 308 5 0 0 mtail [41281413.790392] [54692] 0 54692 16378 179 35 3 0 -1000 sshd [41281413.790394] [58189] 0 58189 160951 23329 85 6 0 0 metricbeat [41281413.790396] [94339] 120 94339 28902 5627 18 4 0 0 prometheus-node [41281413.790398] [119748] 33 119748 11319 203 25 3 0 0 systemd [41281413.790401] [119756] 33 119756 57761 1030 46 3 0 0 (sd-pam) [41281413.790403] [24499] 0 24499 56331 2402 42 3 0 0 fail2ban-server [41281413.790405] [45425] 992 45425 7073 173 18 3 0 0 tmux [41281413.790407] [45431] 992 45431 4987 111 14 3 0 0 bash [41281413.790410] [46660] 0 46660 18568 195 39 3 0 0 sudo [41281413.790412] [46784] 0 46784 18366 155 39 3 0 0 su [41281413.790414] [46785] 0 46785 6403 1548 17 3 0 0 bash [41281413.790416] [47896] 0 47896 10634 216 23 3 0 -1000 systemd-udevd [41281413.790419] [74729] 113 74729 1658892 47321 156 4 0 -900 postgres [41281413.790421] [75151] 100 75151 6718 49 17 3 0 0 uuidd [41281413.790423] [75603] 119 75603 27508 156 26 3 0 0 ntpd [41281413.790426] [77530] 111 77530 16268 1235 30 3 0 0 snmpd [41281413.790428] [84296] 0 84296 69271 198 38 3 0 0 polkitd [41281413.790430] [85771] 109 85771 12788 2295 26 3 0 0 unbound [41281413.790433] [86467] 0 86467 17539 3045 39 4 0 0 python3 [41281413.790435] [114474] 0 114474 57072 4253 50 3 0 0 python3 [41281413.790437] [ 379] 108 379 13449 194 28 4 0 0 exim4 [41281413.790439] [29358] 0 29358 9209 195 22 3 0 0 systemd [41281413.790442] [29360] 0 29360 20768 891 42 3 0 0 (sd-pam) [41281413.790444] [32195] 0 32195 6779 424 18 3 0 0 screen [41281413.790446] [32198] 0 32198 6383 1526 18 3 0 0 bash [41281413.790448] [37112] 0 37112 6443 1591 18 3 0 0 bash [41281413.790451] [55742] 0 55742 184941 6924 64 5 0 0 arvados-dispatc [41281413.790453] [23978] 0 23978 50349 1844 64 5 0 0 nginx [41281413.790455] [24032] 0 24032 248262 4426 69 6 0 0 arvados-control [41281413.790457] [24112] 1001 24112 138789 2447 40 5 0 0 arvados-git-htt [41281413.790459] [125340] 0 125340 113548 851 62 3 0 -1000 PassengerAgent [41281413.790462] [125344] 0 125344 1105525 4984 161 7 0 0 PassengerAgent [41281413.790464] [125373] 33 125373 54944 7860 72 5 0 0 nginx [41281413.790466] [125374] 33 125374 54816 7937 71 5 0 0 nginx [41281413.790469] [125375] 33 125375 54816 7460 71 5 0 0 nginx [41281413.790471] [125376] 33 125376 54880 7927 71 5 0 0 nginx [41281413.790473] [125491] 33 125491 862943 653473 1386 7 0 0 ruby [41281413.790475] [125511] 33 125511 407946 249227 592 6 0 0 ruby [41281413.790478] [125531] 33 125531 499564 298005 686 5 0 0 ruby [41281413.790480] [125551] 33 125551 241906 153800 402 4 0 0 ruby [41281413.790482] [125571] 33 125571 269716 180951 455 4 0 0 ruby [41281413.790484] [125591] 33 125591 161321 76145 249 3 0 0 ruby [41281413.790486] [125611] 33 125611 155356 58605 222 3 0 0 ruby [41281413.790488] [ 4145] 33 4145 139349 54161 203 3 0 0 ruby [41281413.790490] [61001] 33 61001 139333 53912 206 3 0 0 ruby [41281413.790493] [114383] 33 114383 139321 43511 184 3 0 0 ruby [41281413.790495] [123017] 33 123017 138890 40540 178 3 0 0 ruby [41281413.790497] [ 1136] 33 1136 122266 32262 163 4 0 0 ruby [41281413.790499] [ 1168] 33 1168 122547 30632 158 4 0 0 ruby [41281413.790501] [ 1189] 33 1189 138955 40408 177 4 0 0 ruby [41281413.790503] [ 4447] 33 4447 122338 30773 158 4 0 0 ruby [41281413.790505] [ 4481] 33 4481 122490 40511 176 4 0 0 ruby [41281413.790507] [ 5936] 33 5936 139027 37764 170 4 0 0 ruby [41281413.790509] [ 9156] 33 9156 105769 30061 155 4 0 0 ruby [41281413.790511] [ 9182] 33 9182 105793 31844 158 4 0 0 ruby [41281413.790513] [ 9204] 33 9204 122458 30853 157 4 0 0 ruby [41281413.790515] [ 9829] 33 9829 122739 31194 157 4 0 0 ruby [41281413.790517] [ 9855] 33 9855 139532 42058 178 4 0 0 ruby [41281413.790520] [ 9877] 33 9877 105889 28246 151 4 0 0 ruby [41281413.790522] [13229] 33 13229 122682 49170 192 4 0 0 ruby [41281413.790524] [13251] 33 13251 105937 29503 154 4 0 0 ruby [41281413.790526] [13272] 33 13272 122730 42482 180 4 0 0 ruby [41281413.790528] [13297] 33 13297 122883 29914 155 4 0 0 ruby [41281413.790530] [13317] 33 13317 106009 23895 143 4 0 0 ruby [41281413.790532] [13342] 33 13342 106033 29062 154 4 0 0 ruby [41281413.790534] [13362] 33 13362 106057 31936 159 4 0 0 ruby [41281413.790536] [13382] 33 13382 122850 39516 173 4 0 0 ruby [41281413.790538] [13404] 33 13404 122489 37348 169 4 0 0 ruby [41281413.790540] [13424] 33 13424 106129 30030 155 4 0 0 ruby [41281413.790542] [13451] 33 13451 106153 29859 154 4 0 0 ruby [41281413.790547] [14038] 33 14038 106177 24131 143 4 0 0 ruby [41281413.790557] [14058] 33 14058 106201 29057 154 4 0 0 ruby [41281413.790559] [14080] 33 14080 106225 29455 153 4 0 0 ruby [41281413.790562] [14100] 33 14100 106249 24038 143 4 0 0 ruby [41281413.790564] [14120] 33 14120 106273 24170 143 4 0 0 ruby [41281413.790566] [14142] 33 14142 106297 24309 144 4 0 0 ruby [41281413.790568] [14163] 33 14163 123296 50440 196 4 0 0 ruby [41281413.790570] [14917] 33 14917 124237 51748 198 4 0 0 ruby [41281413.790573] [14941] 33 14941 106369 29158 153 4 0 0 ruby [41281413.790575] [14963] 33 14963 123368 41137 177 4 0 0 ruby [41281413.790577] [14991] 33 14991 123315 24117 145 4 0 0 ruby [41281413.790579] [15011] 33 15011 106441 32097 159 4 0 0 ruby [41281413.790581] [15578] 33 15578 106465 24986 146 4 0 0 ruby [41281413.790583] [15599] 33 15599 122873 38191 171 4 0 0 ruby [41281413.790585] [19209] 33 19209 106513 23883 144 4 0 0 ruby [41281413.790588] [19229] 33 19229 106537 24186 144 4 0 0 ruby [41281413.790590] [19262] 33 19262 106561 23846 144 4 0 0 ruby [41281413.790592] [19824] 33 19824 123483 24194 146 4 0 0 ruby [41281413.790594] [19846] 33 19846 106609 23713 144 4 0 0 ruby [41281413.790596] [19866] 33 19866 106633 26806 149 4 0 0 ruby [41281413.790599] [19890] 33 19890 106657 29083 154 4 0 0 ruby [41281413.790601] [20475] 33 20475 106681 23711 144 4 0 0 ruby [41281413.790603] [20497] 33 20497 106705 23910 144 4 0 0 ruby [41281413.790605] [20526] 33 20526 106729 24162 144 4 0 0 ruby [41281413.790607] [25020] 33 25020 123522 40005 176 4 0 0 ruby [41281413.790609] [25040] 33 25040 106777 24177 144 4 0 0 ruby [41281413.790611] [25064] 33 25064 106801 29335 154 4 0 0 ruby [41281413.790613] [25088] 33 25088 123800 42109 180 4 0 0 ruby [41281413.790615] [25108] 33 25108 106849 24145 145 4 0 0 ruby [41281413.790617] [25133] 33 25133 106873 23642 144 4 0 0 ruby [41281413.790619] [25163] 33 25163 106897 29665 156 4 0 0 ruby [41281413.790622] [30635] 33 30635 106921 31701 159 4 0 0 ruby [41281413.790624] [30659] 33 30659 106945 29511 154 4 0 0 ruby [41281413.790626] [36384] 33 36384 106051 23773 143 4 0 0 ruby [41281413.790628] [42607] 33 42607 105621 24474 144 3 0 0 ruby [41281413.790630] [47378] 33 47378 105645 24437 145 3 0 0 ruby [41281413.790632] [47398] 33 47398 105669 24332 144 3 0 0 ruby [41281413.790634] [47418] 33 47418 105693 36889 169 3 0 0 ruby [41281413.790636] [47441] 33 47441 105717 29512 154 3 0 0 ruby [41281413.790638] [48099] 33 48099 105741 24214 145 3 0 0 ruby [41281413.790640] [53071] 33 53071 105765 24050 144 3 0 0 ruby [41281413.790643] [53091] 33 53091 105789 30481 156 3 0 0 ruby [41281413.790645] [53112] 33 53112 105813 23690 143 3 0 0 ruby [41281413.790647] [78574] 33 78574 105633 23664 144 3 0 0 ruby [41281413.790649] [89469] 33 89469 106051 23667 141 3 0 0 ruby [41281413.790651] [100327] 33 100327 105602 23217 142 4 0 0 ruby [41281413.790654] [108407] 0 108407 220945 9272 87 6 0 0 arvados-ws [41281413.790656] [108418] 33 108418 105626 23574 143 4 0 0 ruby [41281413.790659] [108452] 33 108452 105650 23606 143 4 0 0 ruby [41281413.790661] [125517] 33 125517 105608 24389 145 3 0 0 ruby [41281413.790663] [130918] 33 130918 106054 29552 156 4 0 0 ruby [41281413.790665] [39275] 33 39275 106050 23777 146 4 0 0 ruby [41281413.790667] [98410] 33 98410 105624 23295 142 4 0 0 ruby [41281413.790669] [98878] 33 98878 105648 23480 142 4 0 0 ruby [41281413.790672] [98898] 33 98898 105672 22987 142 4 0 0 ruby [41281413.790674] [98919] 33 98919 105696 23129 142 4 0 0 ruby [41281413.790676] [103762] 33 103762 105720 23664 143 4 0 0 ruby [41281413.790678] [103784] 33 103784 105744 23482 144 4 0 0 ruby [41281413.790680] [103808] 33 103808 105768 23708 143 4 0 0 ruby [41281413.790682] [104432] 33 104432 105792 22991 142 4 0 0 ruby [41281413.790685] [104454] 33 104454 105816 23601 143 4 0 0 ruby [41281413.790687] [104480] 33 104480 105840 23531 144 4 0 0 ruby [41281413.790689] [105011] 33 105011 105864 22981 142 4 0 0 ruby [41281413.790691] [105033] 33 105033 105888 23097 142 4 0 0 ruby [41281413.790693] [105555] 33 105555 105912 23460 143 4 0 0 ruby [41281413.790695] [105580] 33 105580 105936 23187 142 4 0 0 ruby [41281413.790697] [105600] 33 105600 105960 22784 142 4 0 0 ruby [41281413.790699] [105620] 33 105620 105984 23417 142 4 0 0 ruby [41281413.790701] [105640] 33 105640 106008 23477 143 4 0 0 ruby [41281413.790703] [105664] 33 105664 106032 23211 142 4 0 0 ruby [41281413.790705] [105684] 33 105684 106056 23384 142 4 0 0 ruby [41281413.790707] [105704] 33 105704 106080 22966 142 4 0 0 ruby [41281413.790709] [105724] 33 105724 106104 23321 142 4 0 0 ruby [41281413.790711] [105744] 33 105744 106128 23352 143 4 0 0 ruby [41281413.790713] [105764] 33 105764 106152 23423 142 4 0 0 ruby [41281413.790715] [105785] 33 105785 106176 23447 142 4 0 0 ruby [41281413.790717] [109698] 33 109698 106200 22760 141 4 0 0 ruby [41281413.790719] [109724] 33 109724 106224 23224 142 4 0 0 ruby [41281413.790721] [110855] 33 110855 106248 22580 141 4 0 0 ruby [41281413.790723] [110876] 33 110876 106272 22576 141 4 0 0 ruby [41281413.790726] [110898] 33 110898 106296 22911 141 4 0 0 ruby [41281413.790728] [74946] 110 74946 56099 240 21 3 0 0 munged [41281413.790730] [120461] 0 120461 2516410 948342 4439 14 0 0 keep-balance [41281413.790732] [11975] 0 11975 95893 27888 159 3 0 0 ruby [41281413.790734] [12284] 113 12284 1658899 676068 1639 9 0 0 postgres [41281413.790736] [12285] 113 12285 1658868 18548 227 8 0 0 postgres [41281413.790738] [12286] 113 12286 1658868 4662 70 4 0 0 postgres [41281413.790741] [12287] 113 12287 1659057 986 71 4 0 0 postgres [41281413.790743] [12288] 113 12288 37176 473 58 3 0 0 postgres [41281413.790745] [12291] 113 12291 1674615 1074913 2756 9 0 0 postgres [41281413.790747] [12303] 113 12303 1659721 56448 222 4 0 0 postgres [41281413.790749] [12381] 113 12381 1659334 3559 76 4 0 0 postgres [41281413.790751] [12424] 113 12424 1664151 776153 2488 9 0 0 postgres [41281413.790753] [12754] 113 12754 1664092 886495 2684 9 0 0 postgres [41281413.790755] [13772] 113 13772 1661049 449286 2061 9 0 0 postgres [41281413.790757] [13773] 113 13773 1663788 381010 2025 9 0 0 postgres [41281413.790759] [26394] 113 26394 1661148 1343913 3143 9 0 0 postgres [41281413.790761] [44715] 113 44715 1674001 818750 2486 9 0 0 postgres [41281413.790764] [71266] 113 71266 1665039 591880 2285 9 0 0 postgres [41281413.790766] [68653] 113 68653 1663885 485969 2081 9 0 0 postgres [41281413.790768] [42957] 113 42957 1660788 287226 1609 9 0 0 postgres [41281413.790770] [100353] 0 100353 11631 8505 29 3 0 0 atop [41281413.790772] [50644] 113 50644 1661129 320213 1885 9 0 0 postgres [41281413.790774] [49622] 113 49622 1659226 4840 103 5 0 0 postgres [41281413.790777] [69744] 113 69744 1660761 140575 1334 9 0 0 postgres [41281413.790779] [69756] 113 69756 1660643 136057 1357 9 0 0 postgres [41281413.790781] [69764] 113 69764 1660669 200557 1465 9 0 0 postgres [41281413.790783] [74865] 113 74865 1660465 47426 715 8 0 0 postgres [41281413.790785] [74866] 113 74866 1660466 47576 728 9 0 0 postgres [41281413.790788] [74867] 113 74867 1660589 141616 1209 9 0 0 postgres [41281413.790790] [74868] 113 74868 1660472 52461 708 9 0 0 postgres [41281413.790792] [74869] 113 74869 1660503 104379 845 9 0 0 postgres [41281413.790794] [74870] 113 74870 1660421 44286 520 8 0 0 postgres [41281413.790796] [74871] 113 74871 1660627 109718 1258 9 0 0 postgres [41281413.790798] [74872] 113 74872 1660489 96179 753 8 0 0 postgres [41281413.790800] [74873] 113 74873 1660250 32533 500 7 0 0 postgres [41281413.790802] [74874] 113 74874 1660441 94524 735 8 0 0 postgres [41281413.790804] [74875] 113 74875 1660438 42203 639 9 0 0 postgres [41281413.790806] [74876] 113 74876 1659904 92991 713 8 0 0 postgres [41281413.790808] [74877] 113 74877 1660382 83203 587 8 0 0 postgres [41281413.790810] [74878] 113 74878 1659778 89789 704 9 0 0 postgres [41281413.790812] [74879] 113 74879 1660473 51344 626 9 0 0 postgres [41281413.790814] [74880] 113 74880 1660428 30332 488 8 0 0 postgres [41281413.790817] [74881] 113 74881 1659507 88983 684 8 0 0 postgres [41281413.790819] [74882] 113 74882 1660455 93158 716 8 0 0 postgres [41281413.790821] [74883] 113 74883 1660051 33580 553 8 0 0 postgres [41281413.790823] [74884] 113 74884 1660425 36225 563 8 0 0 postgres [41281413.790825] [74886] 113 74886 1660031 33197 552 8 0 0 postgres [41281413.790827] [74887] 113 74887 1660410 89528 661 8 0 0 postgres [41281413.790829] [74888] 113 74888 1660433 36849 562 9 0 0 postgres [41281413.790831] [74889] 113 74889 1660129 32443 525 9 0 0 postgres [41281413.790833] [74890] 113 74890 1660084 32287 540 9 0 0 postgres [41281413.790836] [74891] 113 74891 1660480 86773 634 8 0 0 postgres [41281413.790838] [74892] 113 74892 1659484 82934 600 9 0 0 postgres [41281413.790840] [74893] 113 74893 1659507 84667 625 8 0 0 postgres [41281413.790842] [74895] 113 74895 1659792 32157 514 9 0 0 postgres [41281413.790844] [74896] 113 74896 1659451 27596 469 8 0 0 postgres [41281413.790846] [74905] 113 74905 1660387 32131 513 9 0 0 postgres [41281413.790848] [74906] 113 74906 1660404 30554 495 9 0 0 postgres [41281413.790850] [74911] 113 74911 1659560 42020 494 9 0 0 postgres [41281413.790852] [74923] 113 74923 1659436 31740 516 8 0 0 postgres [41281413.790855] [106972] 113 106972 1659488 30055 499 9 0 0 postgres [41281413.790857] [106983] 113 106983 1659412 28176 477 8 0 0 postgres [41281413.790859] [107001] 113 107001 1659415 25864 451 8 0 0 postgres [41281413.790861] [107004] 113 107004 1659504 37767 484 8 0 0 postgres [41281413.790864] [107007] 113 107007 1659427 27808 468 8 0 0 postgres [41281413.790866] [107012] 113 107012 1659504 34075 464 8 0 0 postgres [41281413.790868] [118908] 113 118908 1660701 211588 1472 9 0 0 postgres [41281413.790871] [123105] 113 123105 1659734 50538 649 9 0 0 postgres [41281413.790873] [ 8278] 113 8278 1659414 26710 453 8 0 0 postgres [41281413.790875] [ 8279] 113 8279 1659413 25273 448 8 0 0 postgres [41281413.790877] [ 8280] 113 8280 1659800 83770 574 8 0 0 postgres [41281413.790879] [ 8281] 113 8281 1659775 36700 481 8 0 0 postgres [41281413.790881] [ 8282] 113 8282 1659414 24442 427 8 0 0 postgres [41281413.790883] [ 8283] 113 8283 1659415 26367 444 8 0 0 postgres [41281413.790885] [ 8284] 113 8284 1659413 24718 427 8 0 0 postgres [41281413.790887] [ 8285] 113 8285 1659447 20535 361 8 0 0 postgres [41281413.790890] [ 8286] 113 8286 1659412 25376 440 8 0 0 postgres [41281413.790892] [ 8287] 113 8287 1659543 25711 424 8 0 0 postgres [41281413.790894] [ 8288] 113 8288 1660044 27888 462 8 0 0 postgres [41281413.790896] [ 8289] 113 8289 1659449 25582 421 9 0 0 postgres [41281413.790898] [ 8290] 113 8290 1659448 24529 403 9 0 0 postgres [41281413.790900] [ 8291] 113 8291 1659451 24777 417 8 0 0 postgres [41281413.790902] [14680] 113 14680 1659413 22930 375 8 0 0 postgres [41281413.790905] [14681] 113 14681 1659370 22245 388 8 0 0 postgres [41281413.790907] [14682] 113 14682 1659503 25043 407 8 0 0 postgres [41281413.790909] [14683] 113 14683 1659944 31793 408 8 0 0 postgres [41281413.790911] [14684] 113 14684 1659426 24072 392 8 0 0 postgres [41281413.790913] [14685] 113 14685 1659415 21448 367 8 0 0 postgres [41281413.790915] [14686] 113 14686 1659548 22490 366 8 0 0 postgres [41281413.790917] [27642] 113 27642 1659414 22425 366 8 0 0 postgres [41281413.790919] [27732] 113 27732 1659836 23151 371 8 0 0 postgres [41281413.790921] [27733] 113 27733 1659413 21864 358 8 0 0 postgres [41281413.790923] [27734] 113 27734 1659449 20717 344 8 0 0 postgres [41281413.790925] [27735] 113 27735 1659494 76190 478 8 0 0 postgres [41281413.790927] [27738] 113 27738 1659782 22816 361 8 0 0 postgres [41281413.790929] [27739] 113 27739 1659487 22599 378 9 0 0 postgres [41281413.790932] [27741] 113 27741 1659493 23440 378 8 0 0 postgres [41281413.790934] [27742] 113 27742 1659413 21402 358 9 0 0 postgres [41281413.790936] [27744] 113 27744 1659415 19779 342 8 0 0 postgres [41281413.790938] [27745] 113 27745 1659450 21344 355 9 0 0 postgres [41281413.790940] [27746] 113 27746 1659413 19472 331 8 0 0 postgres [41281413.790942] [27747] 113 27747 1659461 73594 446 8 0 0 postgres [41281413.790944] [27748] 113 27748 1659473 20453 338 8 0 0 postgres [41281413.790946] [27749] 113 27749 1659378 17115 313 9 0 0 postgres [41281413.790948] [27754] 113 27754 1659415 17671 320 8 0 0 postgres [41281413.790951] [55706] 113 55706 1659415 17991 321 8 0 0 postgres [41281413.790953] [55707] 113 55707 1659371 16905 298 8 0 0 postgres [41281413.790955] [55708] 113 55708 1659450 17453 307 8 0 0 postgres [41281413.790957] [55709] 113 55709 1659448 18955 322 9 0 0 postgres [41281413.790959] [55720] 113 55720 1659497 71908 425 8 0 0 postgres [41281413.790961] [55721] 113 55721 1659379 16762 291 8 0 0 postgres [41281413.790963] [58775] 113 58775 1659371 15408 283 8 0 0 postgres [41281413.790965] [58776] 113 58776 1659451 17172 295 8 0 0 postgres [41281413.790967] [58777] 113 58777 1660036 18155 309 8 0 0 postgres [41281413.790970] [58780] 113 58780 1659415 17548 301 8 0 0 postgres [41281413.790972] [58781] 113 58781 1659415 16894 280 8 0 0 postgres [41281413.790974] [58782] 113 58782 1659379 14943 284 8 0 0 postgres [41281413.790976] [58783] 113 58783 1659426 15840 272 8 0 0 postgres [41281413.790978] [58784] 113 58784 1659502 19639 322 8 0 0 postgres [41281413.790980] [58785] 113 58785 1659371 14059 258 8 0 0 postgres [41281413.790983] [58786] 113 58786 1659415 17002 284 8 0 0 postgres [41281413.790985] [58787] 113 58787 1659427 16608 285 8 0 0 postgres [41281413.790987] [58788] 113 58788 1659420 14362 260 8 0 0 postgres [41281413.790989] [58789] 113 58789 1659371 14887 254 8 0 0 postgres [41281413.790991] [58790] 113 58790 1659371 14661 265 8 0 0 postgres [41281413.790993] [58791] 113 58791 1659486 17840 287 8 0 0 postgres [41281413.790995] [58792] 113 58792 1659415 14966 262 9 0 0 postgres [41281413.790997] [58793] 113 58793 1659415 15452 264 8 0 0 postgres [41281413.790999] [58796] 113 58796 1659370 13590 262 8 0 0 postgres [41281413.791001] [58797] 113 58797 1659415 14648 246 7 0 0 postgres [41281413.791003] [58798] 113 58798 1659415 16590 283 9 0 0 postgres [41281413.791005] [58799] 113 58799 1659413 14926 253 7 0 0 postgres [41281413.791008] [58804] 113 58804 1659415 15253 249 7 0 0 postgres [41281413.791010] [58805] 113 58805 1659379 12945 239 8 0 0 postgres [41281413.791012] [58808] 113 58808 1659417 14594 254 8 0 0 postgres [41281413.791014] [58809] 113 58809 1659370 11944 224 7 0 0 postgres [41281413.791016] [58810] 113 58810 1659371 12352 238 8 0 0 postgres [41281413.791018] [58811] 113 58811 1659455 12615 223 7 0 0 postgres [41281413.791020] [58930] 33 58930 123946 49477 194 3 0 0 ruby [41281413.791022] [59956] 113 59956 1659397 18790 283 7 0 0 postgres [41281413.791025] [59972] 113 59972 1659777 22046 238 7 0 0 postgres [41281413.791027] [59975] 33 59975 106078 22808 141 3 0 0 ruby [41281413.791029] [59994] 113 59994 1659455 12627 222 7 0 0 postgres [41281413.791031] [59995] 33 59995 106102 22839 141 3 0 0 ruby [41281413.791033] [60019] 113 60019 1659455 12450 218 7 0 0 postgres [41281413.791035] [60020] 33 60020 106126 22871 141 3 0 0 ruby [41281413.791038] [60041] 113 60041 1659455 13297 225 8 0 0 postgres [41281413.791040] [60042] 33 60042 106150 23368 142 3 0 0 ruby [41281413.791042] [60061] 113 60061 1659455 11118 202 7 0 0 postgres [41281413.791044] [60062] 33 60062 106174 22761 140 3 0 0 ruby [41281413.791046] [60081] 113 60081 1659455 11568 222 7 0 0 postgres [41281413.791048] [60084] 33 60084 106198 23289 141 3 0 0 ruby [41281413.791050] [60103] 113 60103 1659455 11483 212 7 0 0 postgres [41281413.791052] [60104] 33 60104 106222 22700 141 3 0 0 ruby [41281413.791055] [60123] 113 60123 1659455 11360 207 7 0 0 postgres [41281413.791057] [60125] 33 60125 106246 22754 140 3 0 0 ruby [41281413.791059] [63460] 113 63460 1659455 9926 190 7 0 0 postgres [41281413.791061] [63461] 33 63461 106270 23065 141 3 0 0 ruby [41281413.791063] [63481] 113 63481 1659455 9901 187 7 0 0 postgres [41281413.791065] [63483] 33 63483 106294 22619 140 3 0 0 ruby [41281413.791067] [63504] 113 63504 1659455 9143 181 7 0 0 postgres [41281413.791069] [63505] 33 63505 106318 22629 140 3 0 0 ruby [41281413.791072] [63524] 113 63524 1659455 9005 183 7 0 0 postgres [41281413.791074] [63525] 33 63525 106342 22588 140 3 0 0 ruby [41281413.791076] [63547] 113 63547 1659455 9668 183 7 0 0 postgres [41281413.791078] [63550] 33 63550 106366 22592 140 3 0 0 ruby [41281413.791080] [63571] 113 63571 1659455 9730 182 7 0 0 postgres [41281413.791083] [63396] 0 63396 17878 156 37 3 0 0 cron [41281413.791085] [63399] 0 63399 1125 141 8 3 0 0 sh [41281413.791095] [63400] 0 63400 3835 1445 12 3 0 0 bash [41281413.791097] [63585] 0 63585 42443 5701 52 3 0 0 ruby [41281413.791099] [64651] 0 64651 16377 591 36 3 0 0 sshd [41281413.791101] [64656] 104 64656 16377 178 34 3 0 0 sshd [41281413.791103] [68699] 0 68699 5558 1556 16 3 0 0 bash [41281413.791106] [68768] 0 68768 4938 477 14 3 0 0 check_mk_agent [41281413.791108] [68900] 0 68900 5543 1525 16 3 0 0 check-compute-n [41281413.791110] [68908] 0 68908 17878 156 37 3 0 0 cron [41281413.791112] [68915] 0 68915 1125 138 8 3 0 0 sh [41281413.791114] [68916] 0 68916 1125 158 8 3 0 0 update-switchya [41281413.791117] [68924] 0 68924 3835 1466 12 3 0 0 bash [41281413.791119] [68963] 0 68963 5558 1278 15 3 0 0 bash [41281413.791121] [68964] 0 68964 5558 1309 15 3 0 0 bash [41281413.791123] [68968] 0 68968 4816 169 13 3 0 0 ruby [41281413.791125] [68974] 0 68974 16192 2987 36 3 0 0 apport [41281413.791127] [69076] 0 69076 9706 1579 21 3 0 0 ruby [41281413.791129] [69084] 0 69084 9963 1291 21 3 0 0 ruby [41281413.791131] [69088] 0 69088 1125 83 7 3 0 0 sh [41281413.791133] [69089] 0 69089 6962 128 14 3 0 0 ssh [41281413.791135] Out of memory: Kill process 26394 (postgres) score 187 or sacrifice child [41281413.804297] Killed process 26394 (postgres) total-vm:6644592kB, anon-rss:8312kB, file-rss:5367340kB
Updated by Peter Amstutz about 5 years ago
- Target version changed from 2019-10-09 Sprint to 2019-10-23 Sprint
Updated by Peter Amstutz about 5 years ago
15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f
- Catch exception from get_current_container in runtime_status_update
- Add test
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/
Updated by Peter Amstutz about 5 years ago
- Status changed from New to In Progress
Updated by Eric Biagiotti about 5 years ago
Peter Amstutz wrote:
15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f
- Catch exception from get_current_container in runtime_status_update
- Add test
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/
Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.
Updated by Peter Amstutz about 5 years ago
Eric Biagiotti wrote:
Peter Amstutz wrote:
15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f
- Catch exception from get_current_container in runtime_status_update
- Add test
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/
Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.
15694-cwl-hang @ 800b63d7cad00988778d0610294c305f3facac6a
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1584/
Updated by Eric Biagiotti about 5 years ago
Peter Amstutz wrote:
Eric Biagiotti wrote:
Peter Amstutz wrote:
15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f
- Catch exception from get_current_container in runtime_status_update
- Add test
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/
Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.
15694-cwl-hang @ 800b63d7cad00988778d0610294c305f3facac6a
https://ci.curoverse.com/view/Developer/job/developer-run-tests/1584/
Thanks, assuming the tests pass, this LGTM.
Updated by Peter Amstutz about 5 years ago
- Status changed from In Progress to Resolved
Applied in changeset arvados|1aa2903df6165ecc8164c40957f282d5f7174b21.