Bug #15694

[CWL] workflow hangs after getting 502 Bad Gateway

Added by Peter Amstutz 9 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/09/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

The workflow status polling thread got "502 Bad Gateway", the workflow runner stopped monitoring/submitting jobs but didn't fail either, it hung:

2019-10-07T09:13:21.337649988Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:21.337649988Z arvados.cwl-runner ERROR: Error checking states on API server: %s
2019-10-07T09:13:21.337649988Z Traceback (most recent call last):
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 369, in poll_states
2019-10-07T09:13:21.337649988Z     proc_states = table.list(filters=[["uuid", "in", page]]).execute(num_retries=self.num_retries)
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
2019-10-07T09:13:21.337649988Z     return wrapped(*args, **kwargs)
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/http.py", line 835, in execute
2019-10-07T09:13:21.337649988Z     method=str(self.method), body=self.body, headers=self.headers)
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/googleapiclient/http.py", line 157, in _retry_request
2019-10-07T09:13:21.337649988Z     resp.status if resp else exception)
2019-10-07T09:13:21.337649988Z   File "/usr/lib/python2.7/logging/__init__.py", line 1179, in warning
2019-10-07T09:13:21.337649988Z     self._log(WARNING, msg, args, **kwargs)
2019-10-07T09:13:21.337649988Z   File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
2019-10-07T09:13:21.337649988Z     self.handle(record)
2019-10-07T09:13:21.337649988Z   File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
2019-10-07T09:13:21.337649988Z     self.callHandlers(record)
2019-10-07T09:13:21.337649988Z   File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
2019-10-07T09:13:21.337649988Z     hdlr.handle(record)
2019-10-07T09:13:21.337649988Z   File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
2019-10-07T09:13:21.337649988Z     self.emit(record)
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit
2019-10-07T09:13:21.337649988Z     "%s: %s" % (record.name, record.getMessage())
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update
2019-10-07T09:13:21.337649988Z     current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger)
2019-10-07T09:13:21.337649988Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container
2019-10-07T09:13:21.337649988Z     raise e
2019-10-07T09:13:21.337649988Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:29.060662282Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:29.060662282Z arvados.cwl-runner ERROR: Fatal error in state polling thread.
2019-10-07T09:13:29.060662282Z Traceback (most recent call last):
2019-10-07T09:13:29.060662282Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 371, in poll_states
2019-10-07T09:13:29.060662282Z     logger.exception("Error checking states on API server: %s")
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 1200, in exception
2019-10-07T09:13:29.060662282Z     self.error(msg, *args, **kwargs)
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error
2019-10-07T09:13:29.060662282Z     self._log(ERROR, msg, args, **kwargs)
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
2019-10-07T09:13:29.060662282Z     self.handle(record)
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
2019-10-07T09:13:29.060662282Z     self.callHandlers(record)
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
2019-10-07T09:13:29.060662282Z     hdlr.handle(record)
2019-10-07T09:13:29.060662282Z   File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
2019-10-07T09:13:29.060662282Z     self.emit(record)
2019-10-07T09:13:29.060662282Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit
2019-10-07T09:13:29.060662282Z     "%s: %s" % (record.name, record.getMessage())
2019-10-07T09:13:29.060662282Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update
2019-10-07T09:13:29.060662282Z     current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger)
2019-10-07T09:13:29.060662282Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container
2019-10-07T09:13:29.060662282Z     raise e
2019-10-07T09:13:29.060662282Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:43.365414753Z arvados.cwl-runner INFO: Getting current container: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:43.365414753Z Exception in thread Thread-22:
2019-10-07T09:13:43.365414753Z Traceback (most recent call last):
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
2019-10-07T09:13:43.365414753Z     self.run()
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/threading.py", line 754, in run
2019-10-07T09:13:43.365414753Z     self.__target(*self.__args, **self.__kwargs)
2019-10-07T09:13:43.365414753Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 386, in poll_states
2019-10-07T09:13:43.365414753Z     logger.exception("Fatal error in state polling thread.")
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 1200, in exception
2019-10-07T09:13:43.365414753Z     self.error(msg, *args, **kwargs)
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 1193, in error
2019-10-07T09:13:43.365414753Z     self._log(ERROR, msg, args, **kwargs)
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 1286, in _log
2019-10-07T09:13:43.365414753Z     self.handle(record)
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 1296, in handle
2019-10-07T09:13:43.365414753Z     self.callHandlers(record)
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 1336, in callHandlers
2019-10-07T09:13:43.365414753Z     hdlr.handle(record)
2019-10-07T09:13:43.365414753Z   File "/usr/lib/python2.7/logging/__init__.py", line 759, in handle
2019-10-07T09:13:43.365414753Z     self.emit(record)
2019-10-07T09:13:43.365414753Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 86, in emit
2019-10-07T09:13:43.365414753Z     "%s: %s" % (record.name, record.getMessage())
2019-10-07T09:13:43.365414753Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 259, in runtime_status_update
2019-10-07T09:13:43.365414753Z     current = arvados_cwl.util.get_current_container(self.api, self.num_retries, logger)
2019-10-07T09:13:43.365414753Z   File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/util.py", line 34, in get_current_container
2019-10-07T09:13:43.365414753Z     raise e
2019-10-07T09:13:43.365414753Z ApiError: <HttpError 502 when requesting https://qr1hi.arvadosapi.com/arvados/v1/containers/current?alt=json returned "Bad Gateway">
2019-10-07T09:13:43.365414753Z 
nginx_error.log (6.27 MB) nginx_error.log Nico César, 10/07/2019 03:15 PM

Subtasks

Task #15702: Review 15694-cwl-hangResolvedPeter Amstutz

Associated revisions

Revision 1aa2903d
Added by Peter Amstutz 9 months ago

Merge branch '15694-cwl-hang' closes #15694

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 9 months ago

  • Description updated (diff)

#2 Updated by Nico César 9 months ago

Looks like postgres went down.

App 125491 output: WARNING:  terminating connection because of crash of another server process
App 125491 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125491 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125491 output: [ 2019-10-07 09:13:09.4018 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::ConnectionBad: PQconsumeInput() SSL SYSCALL error: EOF detected
App 125491 output: : SELECT  "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)):
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `async_exec'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `block in exec_no_cache'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:590:in `block in log'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract_adapter.rb:583:in `log'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:600:in `exec_no_cache'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:589:in `execute_and_clear'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/postgresql/database_statements.rb:103:in `exec_query'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:373:in `select'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/database_statements.rb:41:in `select_all'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/query_cache.rb:95:in `select_all'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/querying.rb:39:in `find_by_sql'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:706:in `exec_queries'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:583:in `load'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:260:in `records'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation.rb:256:in `to_a'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:563:in `find_nth_with_limit'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:592:in `find_nth_with_limit_and_offset'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:545:in `find_nth'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/relation/finder_methods.rb:122:in `first'
App 125491 output:      from /var/www/arvados-api/current/app/models/api_client_authorization.rb:225:in `validate'
App 125491 output:      from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:51:in `block in call'
App 125491 output:      from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `each'
App 125491 output:      from /var/www/arvados-api/current/app/middlewares/arvados_api_token.rb:48:in `call'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/engine.rb:522:in `call'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `public_send'
App 125491 output:      from /var/www/arvados-api/shared/vendor_bundle/ruby/2.5.0/gems/railties-5.0.7.2/lib/rails/railtie.rb:193:in `method_missing'
App 125491 output:      from /usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
App 125491 output:      from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:149:in `accept_and_process_next_request'
App 125491 output:      from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
App 125491 output:      from /usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'
App 125491 output:      from /usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
[ W 2019-10-07 09:13:09.4531 125344/Tb age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-428041] Sending 502 response: application did not send a complete response
App 125611 output: WARNING:  terminating connection because of crash of another server process
App 125611 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125611 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 4145 output: WARNING:  terminating connection because of crash of another server process
App 125591 output: WARNING:  terminating connection because of crash of another server process
App 4145 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125591 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 4145 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125591 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125531 output: WARNING:  terminating connection because of crash of another server process
App 125531 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125531 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125571 output: WARNING:  terminating connection because of crash of another server process
App 125571 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125571 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125551 output: WARNING:  terminating connection because of crash of another server process
App 125551 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125551 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 61001 output: WARNING:  terminating connection because of crash of another server process
App 61001 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 61001 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 114383 output: WARNING:  terminating connection because of crash of another server process
App 114383 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 114383 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
App 125511 output: WARNING:  terminating connection because of crash of another server process
App 125511 output: DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
App 125511 output: HINT:  In a moment you should be able to reconnect to the database and repeat your command.
[ W 2019-10-07 09:13:09.5670 125344/Te age/Cor/Con/InternalUtils.cpp:96 ]: [Client 4-428042] Sending 502 response: application did not send a complete response
App 125491 output: [ 2019-10-07 09:13:09.5668 125491/0x000000000311dae8(Worker 1) utils.rb ]: *** Exception ActiveRecord::StatementInvalid in Rack application object (PG::UnableToSend: no connection to the server
App 125491 output: : SELECT  "api_client_authorizations".* FROM "api_client_authorizations" WHERE (api_token='2m0ml035ot45ylu116hjw9jjzzv0mmxcbizb4bp6x7uuvao89i' and (expires_at is null or expires_at > CURRENT_TIMESTAMP)) ORDER BY "api_client_authorizations"."id" ASC LIMIT $1) (process 125491, thread 0x000000000311dae8(Worker 1)):
[41281413.790132] keep-balance invoked oom-killer: gfp_mask=0x24280ca, order=0, oom_score_adj=0
[41281413.790136] keep-balance cpuset=/ mems_allowed=0
[41281413.790142] CPU: 6 PID: 120486 Comm: keep-balance Not tainted 4.4.0-104-generic #127-Ubuntu
[41281413.790144] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  06/02/2017
[41281413.790146]  0000000000000286 a01a7fabf763ace9 ffff8802fc497af8 ffffffff813fb523
[41281413.790149]  ffff8802fc497cb0 ffff880249674600 ffff8802fc497b68 ffffffff8120cb7e
[41281413.790152]  0000000000000015 0000000000000000 ffff8807ba714300 ffff8807b9e64600
[41281413.790154] Call Trace:
[41281413.790162]  [<ffffffff813fb523>] dump_stack+0x63/0x90
[41281413.790166]  [<ffffffff8120cb7e>] dump_header+0x5a/0x1c5
[41281413.790171]  [<ffffffff81393b54>] ? apparmor_capable+0xc4/0x1b0
[41281413.790175]  [<ffffffff81193562>] oom_kill_process+0x202/0x3c0
[41281413.790177]  [<ffffffff81193989>] out_of_memory+0x219/0x460
[41281413.790181]  [<ffffffff81199995>] __alloc_pages_slowpath.constprop.88+0x965/0xb00
[41281413.790184]  [<ffffffff81199db6>] __alloc_pages_nodemask+0x286/0x2a0
[41281413.790187]  [<ffffffff811e50bd>] alloc_pages_vma+0xad/0x250
[41281413.790190]  [<ffffffff811c2c5e>] handle_mm_fault+0x110e/0x1820
[41281413.790195]  [<ffffffff8106b577>] __do_page_fault+0x197/0x400
[41281413.790197]  [<ffffffff8106b802>] do_page_fault+0x22/0x30
[41281413.790201]  [<ffffffff818469f8>] page_fault+0x28/0x30
[41281413.790202] Mem-Info:
[41281413.790207] active_anon:5860563 inactive_anon:1043589 isolated_anon:0
                   active_file:611 inactive_file:643 isolated_file:2
                   unevictable:8560 dirty:0 writeback:0 unstable:0
                   slab_reclaimable:37088 slab_unreclaimable:36027
                   mapped:1582583 shmem:1631218 pagetables:116589 bounce:0
                   free:48997 free_pcp:30 free_cma:0
[41281413.790211] Node 0 DMA free:15876kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[41281413.790217] lowmem_reserve[]: 0 417 28110 28110 28110
[41281413.790221] Node 0 DMA32 free:111672kB min:960kB low:1200kB high:1440kB active_anon:298292kB inactive_anon:9296kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:507840kB managed:427136kB mlocked:0kB dirty:0kB writeback:0kB mapped:19392kB shmem:19460kB slab_reclaimable:1260kB slab_unreclaimable:1008kB kernel_stack:176kB pagetables:2604kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? yes
[41281413.790227] lowmem_reserve[]: 0 0 27693 27693 27693
[41281413.790230] Node 0 Normal free:68440kB min:66580kB low:83224kB high:99868kB active_anon:23143960kB inactive_anon:4165060kB active_file:2452kB inactive_file:2576kB unevictable:34240kB isolated(anon):0kB isolated(file):8kB present:28835840kB managed:28358404kB mlocked:34240kB dirty:0kB writeback:0kB mapped:6310940kB shmem:6505412kB slab_reclaimable:147092kB slab_unreclaimable:143100kB kernel_stack:22672kB pagetables:463752kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:34304 all_unreclaimable? yes
[41281413.790236] lowmem_reserve[]: 0 0 0 0 0
[41281413.790239] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15876kB
[41281413.790250] Node 0 DMA32: 238*4kB (UME) 1168*8kB (UME) 1373*16kB (UME) 595*32kB (UME) 163*64kB (UME) 53*128kB (UME) 37*256kB (UME) 16*512kB (UME) 13*1024kB (UME) 2*2048kB (UM) 2*4096kB (M) = 111784kB
[41281413.790278] Node 0 Normal: 728*4kB (UME) 1259*8kB (UME) 1072*16kB (UMEH) 788*32kB (UMEH) 193*64kB (UMEH) 4*128kB (H) 2*256kB (MH) 1*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 69240kB
[41281413.790290] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[41281413.790292] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[41281413.790293] 1633529 total pagecache pages
[41281413.790294] 0 pages in swap cache
[41281413.790296] Swap cache stats: add 0, delete 0, find 0/0
[41281413.790297] Free swap  = 0kB
[41281413.790298] Total swap = 0kB
[41281413.790299] 7339918 pages RAM
[41281413.790300] 0 pages HighMem/MovableOnly
[41281413.790301] 139556 pages reserved
[41281413.790302] 0 pages cma reserved
[41281413.790303] 0 pages hwpoisoned
[41281413.790304] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[41281413.790311] [  446]     0   446     3051     1081      11       3        0             0 haveged
[41281413.790313] [  850]     0   850     4030      213      12       3        0             0 dhclient
[41281413.790315] [  958]     0   958     7548      503      20       4        0             0 systemd-logind
[41281413.790318] [  972]     0   972    17152      523      37       3        0             0 syslog-ng
[41281413.790320] [  974]     0   974    68644      730      37       3        0             0 accounts-daemon
[41281413.790322] [  981]     0   981     7470       56      19       3        0             0 cgmanager
[41281413.790324] [  989]     0   989     6511       52      18       3        0             0 atd
[41281413.790327] [  994]     0   994     1099       23       8       3        0             0 runsvdir
[41281413.790329] [ 1000]     0  1000     6932       73      17       3        0             0 cron
[41281413.790331] [ 1018]     0  1018     1100       23       8       3        0             0 acpid
[41281413.790334] [ 1025]   102  1025    10756      155      25       3        0          -900 dbus-daemon
[41281413.790336] [ 1051]     0  1051     1061       21       7       3        0             0 runsv
[41281413.790338] [ 1052]     0  1052     1061       21       7       3        0             0 runsv
[41281413.790340] [ 1053]     0  1053     1061       21       7       3        0             0 runsv
[41281413.790342] [ 1054]     0  1054     1061       21       7       3        0             0 runsv
[41281413.790344] [ 1067]     0  1067     1097       30       7       3        0             0 svlogd
[41281413.790346] [ 1068]     0  1068     1097       30       7       3        0             0 svlogd
[41281413.790348] [ 1089]     0  1089     1097       30       8       3        0             0 svlogd
[41281413.790350] [ 1122]     0  1122     1097       31       8       3        0             0 svlogd
[41281413.790352] [ 1160]     0  1160     4868       61      16       3        0             0 irqbalance
[41281413.790355] [ 1236]   999  1236    20945     4125      35       6        0             0 consul
[41281413.790357] [ 1426]     0  1426     3764       47      14       3        0             0 xinetd
[41281413.790359] [ 1444]     0  1444     3665       34      13       3        0             0 agetty
[41281413.790361] [ 1521]     0  1521     3619       36      12       3        0             0 agetty
[41281413.790364] [63492]   992 63492    11319      205      27       3        0             0 systemd
[41281413.790366] [63494]   992 63494    41031      680      44       3        0             0 (sd-pam)
[41281413.790368] [63673]   992 63673     6538      197      17       3        0             0 screen
[41281413.790371] [63674]   992 63674     4988      110      14       3        0             0 bash
[41281413.790373] [63703]     0 63703    18569      195      41       3        0             0 sudo
[41281413.790375] [63806]     0 63806    18366      155      40       3        0             0 su
[41281413.790378] [63807]     0 63807     9676     4849      23       3        0             0 bash
[41281413.790380] [48666]  1001 48666    11319      206      25       3        0             0 systemd
[41281413.790382] [48669]  1001 48669    41031      680      44       3        0             0 (sd-pam)
[41281413.790384] [130006]     0 130006   170933     1670     258       4        0             0 systemd-journal
[41281413.790387] [29786]  4047 29786    17384     2854      37       5        0             0 vault
[41281413.790389] [96914]     0 96914   377988    96637     308       5        0             0 mtail
[41281413.790392] [54692]     0 54692    16378      179      35       3        0         -1000 sshd
[41281413.790394] [58189]     0 58189   160951    23329      85       6        0             0 metricbeat
[41281413.790396] [94339]   120 94339    28902     5627      18       4        0             0 prometheus-node
[41281413.790398] [119748]    33 119748    11319      203      25       3        0             0 systemd
[41281413.790401] [119756]    33 119756    57761     1030      46       3        0             0 (sd-pam)
[41281413.790403] [24499]     0 24499    56331     2402      42       3        0             0 fail2ban-server
[41281413.790405] [45425]   992 45425     7073      173      18       3        0             0 tmux
[41281413.790407] [45431]   992 45431     4987      111      14       3        0             0 bash
[41281413.790410] [46660]     0 46660    18568      195      39       3        0             0 sudo
[41281413.790412] [46784]     0 46784    18366      155      39       3        0             0 su
[41281413.790414] [46785]     0 46785     6403     1548      17       3        0             0 bash
[41281413.790416] [47896]     0 47896    10634      216      23       3        0         -1000 systemd-udevd
[41281413.790419] [74729]   113 74729  1658892    47321     156       4        0          -900 postgres
[41281413.790421] [75151]   100 75151     6718       49      17       3        0             0 uuidd
[41281413.790423] [75603]   119 75603    27508      156      26       3        0             0 ntpd
[41281413.790426] [77530]   111 77530    16268     1235      30       3        0             0 snmpd
[41281413.790428] [84296]     0 84296    69271      198      38       3        0             0 polkitd
[41281413.790430] [85771]   109 85771    12788     2295      26       3        0             0 unbound
[41281413.790433] [86467]     0 86467    17539     3045      39       4        0             0 python3
[41281413.790435] [114474]     0 114474    57072     4253      50       3        0             0 python3
[41281413.790437] [  379]   108   379    13449      194      28       4        0             0 exim4
[41281413.790439] [29358]     0 29358     9209      195      22       3        0             0 systemd
[41281413.790442] [29360]     0 29360    20768      891      42       3        0             0 (sd-pam)
[41281413.790444] [32195]     0 32195     6779      424      18       3        0             0 screen
[41281413.790446] [32198]     0 32198     6383     1526      18       3        0             0 bash
[41281413.790448] [37112]     0 37112     6443     1591      18       3        0             0 bash
[41281413.790451] [55742]     0 55742   184941     6924      64       5        0             0 arvados-dispatc
[41281413.790453] [23978]     0 23978    50349     1844      64       5        0             0 nginx
[41281413.790455] [24032]     0 24032   248262     4426      69       6        0             0 arvados-control
[41281413.790457] [24112]  1001 24112   138789     2447      40       5        0             0 arvados-git-htt
[41281413.790459] [125340]     0 125340   113548      851      62       3        0         -1000 PassengerAgent
[41281413.790462] [125344]     0 125344  1105525     4984     161       7        0             0 PassengerAgent
[41281413.790464] [125373]    33 125373    54944     7860      72       5        0             0 nginx
[41281413.790466] [125374]    33 125374    54816     7937      71       5        0             0 nginx
[41281413.790469] [125375]    33 125375    54816     7460      71       5        0             0 nginx
[41281413.790471] [125376]    33 125376    54880     7927      71       5        0             0 nginx
[41281413.790473] [125491]    33 125491   862943   653473    1386       7        0             0 ruby
[41281413.790475] [125511]    33 125511   407946   249227     592       6        0             0 ruby
[41281413.790478] [125531]    33 125531   499564   298005     686       5        0             0 ruby
[41281413.790480] [125551]    33 125551   241906   153800     402       4        0             0 ruby
[41281413.790482] [125571]    33 125571   269716   180951     455       4        0             0 ruby
[41281413.790484] [125591]    33 125591   161321    76145     249       3        0             0 ruby
[41281413.790486] [125611]    33 125611   155356    58605     222       3        0             0 ruby
[41281413.790488] [ 4145]    33  4145   139349    54161     203       3        0             0 ruby
[41281413.790490] [61001]    33 61001   139333    53912     206       3        0             0 ruby
[41281413.790493] [114383]    33 114383   139321    43511     184       3        0             0 ruby
[41281413.790495] [123017]    33 123017   138890    40540     178       3        0             0 ruby
[41281413.790497] [ 1136]    33  1136   122266    32262     163       4        0             0 ruby
[41281413.790499] [ 1168]    33  1168   122547    30632     158       4        0             0 ruby
[41281413.790501] [ 1189]    33  1189   138955    40408     177       4        0             0 ruby
[41281413.790503] [ 4447]    33  4447   122338    30773     158       4        0             0 ruby
[41281413.790505] [ 4481]    33  4481   122490    40511     176       4        0             0 ruby
[41281413.790507] [ 5936]    33  5936   139027    37764     170       4        0             0 ruby
[41281413.790509] [ 9156]    33  9156   105769    30061     155       4        0             0 ruby
[41281413.790511] [ 9182]    33  9182   105793    31844     158       4        0             0 ruby
[41281413.790513] [ 9204]    33  9204   122458    30853     157       4        0             0 ruby
[41281413.790515] [ 9829]    33  9829   122739    31194     157       4        0             0 ruby
[41281413.790517] [ 9855]    33  9855   139532    42058     178       4        0             0 ruby
[41281413.790520] [ 9877]    33  9877   105889    28246     151       4        0             0 ruby
[41281413.790522] [13229]    33 13229   122682    49170     192       4        0             0 ruby
[41281413.790524] [13251]    33 13251   105937    29503     154       4        0             0 ruby
[41281413.790526] [13272]    33 13272   122730    42482     180       4        0             0 ruby
[41281413.790528] [13297]    33 13297   122883    29914     155       4        0             0 ruby
[41281413.790530] [13317]    33 13317   106009    23895     143       4        0             0 ruby
[41281413.790532] [13342]    33 13342   106033    29062     154       4        0             0 ruby
[41281413.790534] [13362]    33 13362   106057    31936     159       4        0             0 ruby
[41281413.790536] [13382]    33 13382   122850    39516     173       4        0             0 ruby
[41281413.790538] [13404]    33 13404   122489    37348     169       4        0             0 ruby
[41281413.790540] [13424]    33 13424   106129    30030     155       4        0             0 ruby
[41281413.790542] [13451]    33 13451   106153    29859     154       4        0             0 ruby
[41281413.790547] [14038]    33 14038   106177    24131     143       4        0             0 ruby
[41281413.790557] [14058]    33 14058   106201    29057     154       4        0             0 ruby
[41281413.790559] [14080]    33 14080   106225    29455     153       4        0             0 ruby
[41281413.790562] [14100]    33 14100   106249    24038     143       4        0             0 ruby
[41281413.790564] [14120]    33 14120   106273    24170     143       4        0             0 ruby
[41281413.790566] [14142]    33 14142   106297    24309     144       4        0             0 ruby
[41281413.790568] [14163]    33 14163   123296    50440     196       4        0             0 ruby
[41281413.790570] [14917]    33 14917   124237    51748     198       4        0             0 ruby
[41281413.790573] [14941]    33 14941   106369    29158     153       4        0             0 ruby
[41281413.790575] [14963]    33 14963   123368    41137     177       4        0             0 ruby
[41281413.790577] [14991]    33 14991   123315    24117     145       4        0             0 ruby
[41281413.790579] [15011]    33 15011   106441    32097     159       4        0             0 ruby
[41281413.790581] [15578]    33 15578   106465    24986     146       4        0             0 ruby
[41281413.790583] [15599]    33 15599   122873    38191     171       4        0             0 ruby
[41281413.790585] [19209]    33 19209   106513    23883     144       4        0             0 ruby
[41281413.790588] [19229]    33 19229   106537    24186     144       4        0             0 ruby
[41281413.790590] [19262]    33 19262   106561    23846     144       4        0             0 ruby
[41281413.790592] [19824]    33 19824   123483    24194     146       4        0             0 ruby
[41281413.790594] [19846]    33 19846   106609    23713     144       4        0             0 ruby
[41281413.790596] [19866]    33 19866   106633    26806     149       4        0             0 ruby
[41281413.790599] [19890]    33 19890   106657    29083     154       4        0             0 ruby
[41281413.790601] [20475]    33 20475   106681    23711     144       4        0             0 ruby
[41281413.790603] [20497]    33 20497   106705    23910     144       4        0             0 ruby
[41281413.790605] [20526]    33 20526   106729    24162     144       4        0             0 ruby
[41281413.790607] [25020]    33 25020   123522    40005     176       4        0             0 ruby
[41281413.790609] [25040]    33 25040   106777    24177     144       4        0             0 ruby
[41281413.790611] [25064]    33 25064   106801    29335     154       4        0             0 ruby
[41281413.790613] [25088]    33 25088   123800    42109     180       4        0             0 ruby
[41281413.790615] [25108]    33 25108   106849    24145     145       4        0             0 ruby
[41281413.790617] [25133]    33 25133   106873    23642     144       4        0             0 ruby
[41281413.790619] [25163]    33 25163   106897    29665     156       4        0             0 ruby
[41281413.790622] [30635]    33 30635   106921    31701     159       4        0             0 ruby
[41281413.790624] [30659]    33 30659   106945    29511     154       4        0             0 ruby
[41281413.790626] [36384]    33 36384   106051    23773     143       4        0             0 ruby
[41281413.790628] [42607]    33 42607   105621    24474     144       3        0             0 ruby
[41281413.790630] [47378]    33 47378   105645    24437     145       3        0             0 ruby
[41281413.790632] [47398]    33 47398   105669    24332     144       3        0             0 ruby
[41281413.790634] [47418]    33 47418   105693    36889     169       3        0             0 ruby
[41281413.790636] [47441]    33 47441   105717    29512     154       3        0             0 ruby
[41281413.790638] [48099]    33 48099   105741    24214     145       3        0             0 ruby
[41281413.790640] [53071]    33 53071   105765    24050     144       3        0             0 ruby
[41281413.790643] [53091]    33 53091   105789    30481     156       3        0             0 ruby
[41281413.790645] [53112]    33 53112   105813    23690     143       3        0             0 ruby
[41281413.790647] [78574]    33 78574   105633    23664     144       3        0             0 ruby
[41281413.790649] [89469]    33 89469   106051    23667     141       3        0             0 ruby
[41281413.790651] [100327]    33 100327   105602    23217     142       4        0             0 ruby
[41281413.790654] [108407]     0 108407   220945     9272      87       6        0             0 arvados-ws
[41281413.790656] [108418]    33 108418   105626    23574     143       4        0             0 ruby
[41281413.790659] [108452]    33 108452   105650    23606     143       4        0             0 ruby
[41281413.790661] [125517]    33 125517   105608    24389     145       3        0             0 ruby
[41281413.790663] [130918]    33 130918   106054    29552     156       4        0             0 ruby
[41281413.790665] [39275]    33 39275   106050    23777     146       4        0             0 ruby
[41281413.790667] [98410]    33 98410   105624    23295     142       4        0             0 ruby
[41281413.790669] [98878]    33 98878   105648    23480     142       4        0             0 ruby
[41281413.790672] [98898]    33 98898   105672    22987     142       4        0             0 ruby
[41281413.790674] [98919]    33 98919   105696    23129     142       4        0             0 ruby
[41281413.790676] [103762]    33 103762   105720    23664     143       4        0             0 ruby
[41281413.790678] [103784]    33 103784   105744    23482     144       4        0             0 ruby
[41281413.790680] [103808]    33 103808   105768    23708     143       4        0             0 ruby
[41281413.790682] [104432]    33 104432   105792    22991     142       4        0             0 ruby
[41281413.790685] [104454]    33 104454   105816    23601     143       4        0             0 ruby
[41281413.790687] [104480]    33 104480   105840    23531     144       4        0             0 ruby
[41281413.790689] [105011]    33 105011   105864    22981     142       4        0             0 ruby
[41281413.790691] [105033]    33 105033   105888    23097     142       4        0             0 ruby
[41281413.790693] [105555]    33 105555   105912    23460     143       4        0             0 ruby
[41281413.790695] [105580]    33 105580   105936    23187     142       4        0             0 ruby
[41281413.790697] [105600]    33 105600   105960    22784     142       4        0             0 ruby
[41281413.790699] [105620]    33 105620   105984    23417     142       4        0             0 ruby
[41281413.790701] [105640]    33 105640   106008    23477     143       4        0             0 ruby
[41281413.790703] [105664]    33 105664   106032    23211     142       4        0             0 ruby
[41281413.790705] [105684]    33 105684   106056    23384     142       4        0             0 ruby
[41281413.790707] [105704]    33 105704   106080    22966     142       4        0             0 ruby
[41281413.790709] [105724]    33 105724   106104    23321     142       4        0             0 ruby
[41281413.790711] [105744]    33 105744   106128    23352     143       4        0             0 ruby
[41281413.790713] [105764]    33 105764   106152    23423     142       4        0             0 ruby
[41281413.790715] [105785]    33 105785   106176    23447     142       4        0             0 ruby
[41281413.790717] [109698]    33 109698   106200    22760     141       4        0             0 ruby
[41281413.790719] [109724]    33 109724   106224    23224     142       4        0             0 ruby
[41281413.790721] [110855]    33 110855   106248    22580     141       4        0             0 ruby
[41281413.790723] [110876]    33 110876   106272    22576     141       4        0             0 ruby
[41281413.790726] [110898]    33 110898   106296    22911     141       4        0             0 ruby
[41281413.790728] [74946]   110 74946    56099      240      21       3        0             0 munged
[41281413.790730] [120461]     0 120461  2516410   948342    4439      14        0             0 keep-balance
[41281413.790732] [11975]     0 11975    95893    27888     159       3        0             0 ruby
[41281413.790734] [12284]   113 12284  1658899   676068    1639       9        0             0 postgres
[41281413.790736] [12285]   113 12285  1658868    18548     227       8        0             0 postgres
[41281413.790738] [12286]   113 12286  1658868     4662      70       4        0             0 postgres
[41281413.790741] [12287]   113 12287  1659057      986      71       4        0             0 postgres
[41281413.790743] [12288]   113 12288    37176      473      58       3        0             0 postgres
[41281413.790745] [12291]   113 12291  1674615  1074913    2756       9        0             0 postgres
[41281413.790747] [12303]   113 12303  1659721    56448     222       4        0             0 postgres
[41281413.790749] [12381]   113 12381  1659334     3559      76       4        0             0 postgres
[41281413.790751] [12424]   113 12424  1664151   776153    2488       9        0             0 postgres
[41281413.790753] [12754]   113 12754  1664092   886495    2684       9        0             0 postgres
[41281413.790755] [13772]   113 13772  1661049   449286    2061       9        0             0 postgres
[41281413.790757] [13773]   113 13773  1663788   381010    2025       9        0             0 postgres
[41281413.790759] [26394]   113 26394  1661148  1343913    3143       9        0             0 postgres
[41281413.790761] [44715]   113 44715  1674001   818750    2486       9        0             0 postgres
[41281413.790764] [71266]   113 71266  1665039   591880    2285       9        0             0 postgres
[41281413.790766] [68653]   113 68653  1663885   485969    2081       9        0             0 postgres
[41281413.790768] [42957]   113 42957  1660788   287226    1609       9        0             0 postgres
[41281413.790770] [100353]     0 100353    11631     8505      29       3        0             0 atop
[41281413.790772] [50644]   113 50644  1661129   320213    1885       9        0             0 postgres
[41281413.790774] [49622]   113 49622  1659226     4840     103       5        0             0 postgres
[41281413.790777] [69744]   113 69744  1660761   140575    1334       9        0             0 postgres
[41281413.790779] [69756]   113 69756  1660643   136057    1357       9        0             0 postgres
[41281413.790781] [69764]   113 69764  1660669   200557    1465       9        0             0 postgres
[41281413.790783] [74865]   113 74865  1660465    47426     715       8        0             0 postgres
[41281413.790785] [74866]   113 74866  1660466    47576     728       9        0             0 postgres
[41281413.790788] [74867]   113 74867  1660589   141616    1209       9        0             0 postgres
[41281413.790790] [74868]   113 74868  1660472    52461     708       9        0             0 postgres
[41281413.790792] [74869]   113 74869  1660503   104379     845       9        0             0 postgres
[41281413.790794] [74870]   113 74870  1660421    44286     520       8        0             0 postgres
[41281413.790796] [74871]   113 74871  1660627   109718    1258       9        0             0 postgres
[41281413.790798] [74872]   113 74872  1660489    96179     753       8        0             0 postgres
[41281413.790800] [74873]   113 74873  1660250    32533     500       7        0             0 postgres
[41281413.790802] [74874]   113 74874  1660441    94524     735       8        0             0 postgres
[41281413.790804] [74875]   113 74875  1660438    42203     639       9        0             0 postgres
[41281413.790806] [74876]   113 74876  1659904    92991     713       8        0             0 postgres
[41281413.790808] [74877]   113 74877  1660382    83203     587       8        0             0 postgres
[41281413.790810] [74878]   113 74878  1659778    89789     704       9        0             0 postgres
[41281413.790812] [74879]   113 74879  1660473    51344     626       9        0             0 postgres
[41281413.790814] [74880]   113 74880  1660428    30332     488       8        0             0 postgres
[41281413.790817] [74881]   113 74881  1659507    88983     684       8        0             0 postgres
[41281413.790819] [74882]   113 74882  1660455    93158     716       8        0             0 postgres
[41281413.790821] [74883]   113 74883  1660051    33580     553       8        0             0 postgres
[41281413.790823] [74884]   113 74884  1660425    36225     563       8        0             0 postgres
[41281413.790825] [74886]   113 74886  1660031    33197     552       8        0             0 postgres
[41281413.790827] [74887]   113 74887  1660410    89528     661       8        0             0 postgres
[41281413.790829] [74888]   113 74888  1660433    36849     562       9        0             0 postgres
[41281413.790831] [74889]   113 74889  1660129    32443     525       9        0             0 postgres
[41281413.790833] [74890]   113 74890  1660084    32287     540       9        0             0 postgres
[41281413.790836] [74891]   113 74891  1660480    86773     634       8        0             0 postgres
[41281413.790838] [74892]   113 74892  1659484    82934     600       9        0             0 postgres
[41281413.790840] [74893]   113 74893  1659507    84667     625       8        0             0 postgres
[41281413.790842] [74895]   113 74895  1659792    32157     514       9        0             0 postgres
[41281413.790844] [74896]   113 74896  1659451    27596     469       8        0             0 postgres
[41281413.790846] [74905]   113 74905  1660387    32131     513       9        0             0 postgres
[41281413.790848] [74906]   113 74906  1660404    30554     495       9        0             0 postgres
[41281413.790850] [74911]   113 74911  1659560    42020     494       9        0             0 postgres
[41281413.790852] [74923]   113 74923  1659436    31740     516       8        0             0 postgres
[41281413.790855] [106972]   113 106972  1659488    30055     499       9        0             0 postgres
[41281413.790857] [106983]   113 106983  1659412    28176     477       8        0             0 postgres
[41281413.790859] [107001]   113 107001  1659415    25864     451       8        0             0 postgres
[41281413.790861] [107004]   113 107004  1659504    37767     484       8        0             0 postgres
[41281413.790864] [107007]   113 107007  1659427    27808     468       8        0             0 postgres
[41281413.790866] [107012]   113 107012  1659504    34075     464       8        0             0 postgres
[41281413.790868] [118908]   113 118908  1660701   211588    1472       9        0             0 postgres
[41281413.790871] [123105]   113 123105  1659734    50538     649       9        0             0 postgres
[41281413.790873] [ 8278]   113  8278  1659414    26710     453       8        0             0 postgres
[41281413.790875] [ 8279]   113  8279  1659413    25273     448       8        0             0 postgres
[41281413.790877] [ 8280]   113  8280  1659800    83770     574       8        0             0 postgres
[41281413.790879] [ 8281]   113  8281  1659775    36700     481       8        0             0 postgres
[41281413.790881] [ 8282]   113  8282  1659414    24442     427       8        0             0 postgres
[41281413.790883] [ 8283]   113  8283  1659415    26367     444       8        0             0 postgres
[41281413.790885] [ 8284]   113  8284  1659413    24718     427       8        0             0 postgres
[41281413.790887] [ 8285]   113  8285  1659447    20535     361       8        0             0 postgres
[41281413.790890] [ 8286]   113  8286  1659412    25376     440       8        0             0 postgres
[41281413.790892] [ 8287]   113  8287  1659543    25711     424       8        0             0 postgres
[41281413.790894] [ 8288]   113  8288  1660044    27888     462       8        0             0 postgres
[41281413.790896] [ 8289]   113  8289  1659449    25582     421       9        0             0 postgres
[41281413.790898] [ 8290]   113  8290  1659448    24529     403       9        0             0 postgres
[41281413.790900] [ 8291]   113  8291  1659451    24777     417       8        0             0 postgres
[41281413.790902] [14680]   113 14680  1659413    22930     375       8        0             0 postgres
[41281413.790905] [14681]   113 14681  1659370    22245     388       8        0             0 postgres
[41281413.790907] [14682]   113 14682  1659503    25043     407       8        0             0 postgres
[41281413.790909] [14683]   113 14683  1659944    31793     408       8        0             0 postgres
[41281413.790911] [14684]   113 14684  1659426    24072     392       8        0             0 postgres
[41281413.790913] [14685]   113 14685  1659415    21448     367       8        0             0 postgres
[41281413.790915] [14686]   113 14686  1659548    22490     366       8        0             0 postgres
[41281413.790917] [27642]   113 27642  1659414    22425     366       8        0             0 postgres
[41281413.790919] [27732]   113 27732  1659836    23151     371       8        0             0 postgres
[41281413.790921] [27733]   113 27733  1659413    21864     358       8        0             0 postgres
[41281413.790923] [27734]   113 27734  1659449    20717     344       8        0             0 postgres
[41281413.790925] [27735]   113 27735  1659494    76190     478       8        0             0 postgres
[41281413.790927] [27738]   113 27738  1659782    22816     361       8        0             0 postgres
[41281413.790929] [27739]   113 27739  1659487    22599     378       9        0             0 postgres
[41281413.790932] [27741]   113 27741  1659493    23440     378       8        0             0 postgres
[41281413.790934] [27742]   113 27742  1659413    21402     358       9        0             0 postgres
[41281413.790936] [27744]   113 27744  1659415    19779     342       8        0             0 postgres
[41281413.790938] [27745]   113 27745  1659450    21344     355       9        0             0 postgres
[41281413.790940] [27746]   113 27746  1659413    19472     331       8        0             0 postgres
[41281413.790942] [27747]   113 27747  1659461    73594     446       8        0             0 postgres
[41281413.790944] [27748]   113 27748  1659473    20453     338       8        0             0 postgres
[41281413.790946] [27749]   113 27749  1659378    17115     313       9        0             0 postgres
[41281413.790948] [27754]   113 27754  1659415    17671     320       8        0             0 postgres
[41281413.790951] [55706]   113 55706  1659415    17991     321       8        0             0 postgres
[41281413.790953] [55707]   113 55707  1659371    16905     298       8        0             0 postgres
[41281413.790955] [55708]   113 55708  1659450    17453     307       8        0             0 postgres
[41281413.790957] [55709]   113 55709  1659448    18955     322       9        0             0 postgres
[41281413.790959] [55720]   113 55720  1659497    71908     425       8        0             0 postgres
[41281413.790961] [55721]   113 55721  1659379    16762     291       8        0             0 postgres
[41281413.790963] [58775]   113 58775  1659371    15408     283       8        0             0 postgres
[41281413.790965] [58776]   113 58776  1659451    17172     295       8        0             0 postgres
[41281413.790967] [58777]   113 58777  1660036    18155     309       8        0             0 postgres
[41281413.790970] [58780]   113 58780  1659415    17548     301       8        0             0 postgres
[41281413.790972] [58781]   113 58781  1659415    16894     280       8        0             0 postgres
[41281413.790974] [58782]   113 58782  1659379    14943     284       8        0             0 postgres
[41281413.790976] [58783]   113 58783  1659426    15840     272       8        0             0 postgres
[41281413.790978] [58784]   113 58784  1659502    19639     322       8        0             0 postgres
[41281413.790980] [58785]   113 58785  1659371    14059     258       8        0             0 postgres
[41281413.790983] [58786]   113 58786  1659415    17002     284       8        0             0 postgres
[41281413.790985] [58787]   113 58787  1659427    16608     285       8        0             0 postgres
[41281413.790987] [58788]   113 58788  1659420    14362     260       8        0             0 postgres
[41281413.790989] [58789]   113 58789  1659371    14887     254       8        0             0 postgres
[41281413.790991] [58790]   113 58790  1659371    14661     265       8        0             0 postgres
[41281413.790993] [58791]   113 58791  1659486    17840     287       8        0             0 postgres
[41281413.790995] [58792]   113 58792  1659415    14966     262       9        0             0 postgres
[41281413.790997] [58793]   113 58793  1659415    15452     264       8        0             0 postgres
[41281413.790999] [58796]   113 58796  1659370    13590     262       8        0             0 postgres
[41281413.791001] [58797]   113 58797  1659415    14648     246       7        0             0 postgres
[41281413.791003] [58798]   113 58798  1659415    16590     283       9        0             0 postgres
[41281413.791005] [58799]   113 58799  1659413    14926     253       7        0             0 postgres
[41281413.791008] [58804]   113 58804  1659415    15253     249       7        0             0 postgres
[41281413.791010] [58805]   113 58805  1659379    12945     239       8        0             0 postgres
[41281413.791012] [58808]   113 58808  1659417    14594     254       8        0             0 postgres
[41281413.791014] [58809]   113 58809  1659370    11944     224       7        0             0 postgres
[41281413.791016] [58810]   113 58810  1659371    12352     238       8        0             0 postgres
[41281413.791018] [58811]   113 58811  1659455    12615     223       7        0             0 postgres
[41281413.791020] [58930]    33 58930   123946    49477     194       3        0             0 ruby
[41281413.791022] [59956]   113 59956  1659397    18790     283       7        0             0 postgres
[41281413.791025] [59972]   113 59972  1659777    22046     238       7        0             0 postgres
[41281413.791027] [59975]    33 59975   106078    22808     141       3        0             0 ruby
[41281413.791029] [59994]   113 59994  1659455    12627     222       7        0             0 postgres
[41281413.791031] [59995]    33 59995   106102    22839     141       3        0             0 ruby
[41281413.791033] [60019]   113 60019  1659455    12450     218       7        0             0 postgres
[41281413.791035] [60020]    33 60020   106126    22871     141       3        0             0 ruby
[41281413.791038] [60041]   113 60041  1659455    13297     225       8        0             0 postgres
[41281413.791040] [60042]    33 60042   106150    23368     142       3        0             0 ruby
[41281413.791042] [60061]   113 60061  1659455    11118     202       7        0             0 postgres
[41281413.791044] [60062]    33 60062   106174    22761     140       3        0             0 ruby
[41281413.791046] [60081]   113 60081  1659455    11568     222       7        0             0 postgres
[41281413.791048] [60084]    33 60084   106198    23289     141       3        0             0 ruby
[41281413.791050] [60103]   113 60103  1659455    11483     212       7        0             0 postgres
[41281413.791052] [60104]    33 60104   106222    22700     141       3        0             0 ruby
[41281413.791055] [60123]   113 60123  1659455    11360     207       7        0             0 postgres
[41281413.791057] [60125]    33 60125   106246    22754     140       3        0             0 ruby
[41281413.791059] [63460]   113 63460  1659455     9926     190       7        0             0 postgres
[41281413.791061] [63461]    33 63461   106270    23065     141       3        0             0 ruby
[41281413.791063] [63481]   113 63481  1659455     9901     187       7        0             0 postgres
[41281413.791065] [63483]    33 63483   106294    22619     140       3        0             0 ruby
[41281413.791067] [63504]   113 63504  1659455     9143     181       7        0             0 postgres
[41281413.791069] [63505]    33 63505   106318    22629     140       3        0             0 ruby
[41281413.791072] [63524]   113 63524  1659455     9005     183       7        0             0 postgres
[41281413.791074] [63525]    33 63525   106342    22588     140       3        0             0 ruby
[41281413.791076] [63547]   113 63547  1659455     9668     183       7        0             0 postgres
[41281413.791078] [63550]    33 63550   106366    22592     140       3        0             0 ruby
[41281413.791080] [63571]   113 63571  1659455     9730     182       7        0             0 postgres
[41281413.791083] [63396]     0 63396    17878      156      37       3        0             0 cron
[41281413.791085] [63399]     0 63399     1125      141       8       3        0             0 sh
[41281413.791095] [63400]     0 63400     3835     1445      12       3        0             0 bash
[41281413.791097] [63585]     0 63585    42443     5701      52       3        0             0 ruby
[41281413.791099] [64651]     0 64651    16377      591      36       3        0             0 sshd
[41281413.791101] [64656]   104 64656    16377      178      34       3        0             0 sshd
[41281413.791103] [68699]     0 68699     5558     1556      16       3        0             0 bash
[41281413.791106] [68768]     0 68768     4938      477      14       3        0             0 check_mk_agent
[41281413.791108] [68900]     0 68900     5543     1525      16       3        0             0 check-compute-n
[41281413.791110] [68908]     0 68908    17878      156      37       3        0             0 cron
[41281413.791112] [68915]     0 68915     1125      138       8       3        0             0 sh
[41281413.791114] [68916]     0 68916     1125      158       8       3        0             0 update-switchya
[41281413.791117] [68924]     0 68924     3835     1466      12       3        0             0 bash
[41281413.791119] [68963]     0 68963     5558     1278      15       3        0             0 bash
[41281413.791121] [68964]     0 68964     5558     1309      15       3        0             0 bash
[41281413.791123] [68968]     0 68968     4816      169      13       3        0             0 ruby
[41281413.791125] [68974]     0 68974    16192     2987      36       3        0             0 apport
[41281413.791127] [69076]     0 69076     9706     1579      21       3        0             0 ruby
[41281413.791129] [69084]     0 69084     9963     1291      21       3        0             0 ruby
[41281413.791131] [69088]     0 69088     1125       83       7       3        0             0 sh
[41281413.791133] [69089]     0 69089     6962      128      14       3        0             0 ssh
[41281413.791135] Out of memory: Kill process 26394 (postgres) score 187 or sacrifice child
[41281413.804297] Killed process 26394 (postgres) total-vm:6644592kB, anon-rss:8312kB, file-rss:5367340kB

#3 Updated by Peter Amstutz 9 months ago

  • Target version changed from 2019-10-09 Sprint to 2019-10-23 Sprint

#4 Updated by Peter Amstutz 9 months ago

  • Assigned To set to Peter Amstutz

#5 Updated by Peter Amstutz 9 months ago

15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f

  • Catch exception from get_current_container in runtime_status_update
  • Add test

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/

#6 Updated by Peter Amstutz 9 months ago

  • Status changed from New to In Progress

#7 Updated by Eric Biagiotti 9 months ago

Peter Amstutz wrote:

15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f

  • Catch exception from get_current_container in runtime_status_update
  • Add test

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/

Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.

#8 Updated by Peter Amstutz 9 months ago

Eric Biagiotti wrote:

Peter Amstutz wrote:

15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f

  • Catch exception from get_current_container in runtime_status_update
  • Add test

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/

Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.

15694-cwl-hang @ 800b63d7cad00988778d0610294c305f3facac6a

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1584/

#9 Updated by Eric Biagiotti 9 months ago

Peter Amstutz wrote:

Eric Biagiotti wrote:

Peter Amstutz wrote:

15694-cwl-hang @ cbdda43d034b07bfeba9f711e4821c5c28c08e7f

  • Catch exception from get_current_container in runtime_status_update
  • Add test

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1581/

Doesn't look like you finished the comment at the top of the test. The comment in the test isn't accurate either. The fail message could be improved as well. Also, the RuntimeError except clause wont catch the Exception type.

15694-cwl-hang @ 800b63d7cad00988778d0610294c305f3facac6a

https://ci.curoverse.com/view/Developer/job/developer-run-tests/1584/

Thanks, assuming the tests pass, this LGTM.

#10 Updated by Peter Amstutz 9 months ago

  • Status changed from In Progress to Resolved

#11 Updated by Peter Amstutz 6 months ago

  • Release set to 22

Also available in: Atom PDF