Arvados

Upon arv-put debug logs inspection, I've seen that 228 blocks failed to be sent to one of the keepstores (all blocks sent to that keepstore failed) so they were sent to an alternative, and if we assume that all blocks are 64MB, the total gives us ~14.6GB of data... almost the amount of RAM the VM has.

I'll be reviewing the PySDK keep client code to see if I can detect any memory leak situation, and also try to set up a simulated environment to reproduce the problem.

Ok, so I've been able to reproduce the issue using arvbox.

Indeed it seems the problem happens when arv-put (or the PySDK keep client) talks directly to keepstore servers, and one of the keepstores is either down or responding a 500 error. The RAM usage starts to creep up until the process gets killed.

I did a similar test with all the keepstores working and the RAM usage keeps below 500 MB using --threads 1.

After some time doing different tests, I got to a fix that I'm yet not sure why it makes a difference:

diff --git a/sdk/python/arvados/keep.py b/sdk/python/arvados/keep.py
index bc43b849c..bd0e5dc1e 100644
--- a/sdk/python/arvados/keep.py
+++ b/sdk/python/arvados/keep.py
@@ -648,7 +648,7 @@ class KeepClient(object):

     class KeepWriterThread(threading.Thread):
-        TaskFailed = RuntimeError()
+        class TaskFailed(RuntimeError): pass

         def __init__(self, queue, data, data_hash, timeout=None):
             super(KeepClient.KeepWriterThread, self).__init__()
@@ -667,7 +667,7 @@ class KeepClient(object):
                 try:
                     locator, copies = self.do_task(service, service_root)
                 except Exception as e:
-                    if e is not self.TaskFailed:
+                    if not isinstance(e, self.TaskFailed):
                         _logger.exception("Exception in KeepWriterThread")
                     self.queue.write_fail(service)
                 else:
@@ -687,7 +687,7 @@ class KeepClient(object):
                                   self.data_hash,
                                   result['status_code'],
                                   result['body'])
-                raise self.TaskFailed
+                raise self.TaskFailed()

             _logger.debug("KeepWriterThread %s succeeded %s+%i %s",
                           str(threading.current_thread()),

The way I was able to reproduce the bug is by using arvbox on a RAM restricted VM (eg: 1.5 GB), shutting down one of the keepstores, and then by accessing to arvbox's shell, used arv-put to upload a dir with many GBs (supposedly with 2X the VM's RAM is enough). Using arv-put from arvbox's shell is important because this happens when accessing to keepstores directly and one of them fails.

Updates at 0ea8f67a0 - branch 17118-arvput-fix
Test run: developer-run-tests: #2236
WB integration test re-run: developer-run-tests-apps-workbench-integration: #2368

• Changes the way custom exception raising is done on KeepWriterThread, as described on the note above.

Following what's described on https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/, I tried to clear the e.__traceback__ object on some places that I saw the exceptions being assigned to a local scoped variable, but didn't see any meaningful memory usage difference: when all keepstores are working, arv-put uses around 560 MB of resident memory, and when one of the keepstore is down, it uses around 800 MB.