Arvados

I have been unable to reproduce this so far. Running on compute34.qr1hi, using the same API token as the original job, I can run this in a Python interpreter, which I believe is functionally equivalent to what the grep script does:

import arvados
cr = arvados.CollectionReader('fee29077095fed2e695100c299f11dc5+2727')
for s in cr.all_streams():
    for f in s.all_files():
        cr = arvados.CollectionReader(f.as_manifest())
        for crfile in cr.all_files():
                for line in crfile.readlines():
                        pass

This runs to completion without raising an exception.

One weird discrepancy I can't figure out is that the line that raises this exception doesn't match what's in the backtrace (305):

compute34.qr1hi:~$ grep -nFC3 "Block not found" /usr/local/lib/python2.7/dist-packages/arvados/keep.py
300-
301-        slot.set(None)
302-        self.cap_cache()
303:        raise arvados.errors.NotFoundError("Block not found: %s" % expect_hash)
304-
305-    def get_url(self, url, headers, expect_hash):
306-        h = httplib2.Http()

To bring this up to date, here is what we have found. Testing consists of storing ~3GB of data in Keep once, and then running repeated GET requests on random blocks, with one or more concurrent clients on the same machine.

• Base memory footprint for Keep when idle is ~5MB.
• After issuing the necessary PUT requests to Keep, the server's memory footprint is 260MB (256MB higher than base).
• With one client running serial GET requests, server memory footprint rises to 516MB (256MB higher than post-PUT).
• Each additional concurrent client adds roughly 128MB to server footprint
• After clients terminate, memory returns to baseline levels after 5-10 minutes.

Curiously, Go's own profiling tools do not account for the RSS reported by the kernel.

Peter has observed that ioutil.ReadAll grows its buffer exponentially, so when reading 64MB of data over the network, it will allocate a 128MB buffer. That appears likely to be partly responsible for bloating memory usage.

Next steps:

• Rewrite to avoid ioutil.ReadAll:
  • Rewrite main.PutBlock to stream data from the network directly to a temporary file instead of an in-memory buffer, checksumming as it goes.
  • Possibly: rewrite main.GetBlock to return a block even if the checksum does not match, and simply log an error if it does. Doing so will allow us to handle GET with a relatively small buffer. It is reasonable to expect the client to perform its own checksum anyway, so it is not necessarily unwise for Keep to return the source data.
• (lower) Understand the discrepancy between "go tool pprof" and the RSS reported by top. My assumption is that this is an artifact of Go's allocator hanging on to this memory so that the next call to make will not require it to go back to malloc(3) but

I did some testing, and this branch looks good to me. Please merge.

For future reference I also experimented with replacing the

defer runtime.GC()

lines by

defer debug.FreeOSMemory()

which seems to lead to (much) lower but much more variable memory use. With 2 GET and 2 PUT clients accessing 2 different blocks with watch -n0.5, I saw a fairly stable RSS of 400M with this branch. When using FreeOSMemory instead, memory use was about half that on average, but much more variable (between 64M and 400M). CPU load seemed roughly 10% higher.