Arvados

Both volume.Get and main.ReadAtMost use ioutil.ReadAll to read all available data. As Peter observed, ReadAll grows a buffer exponentially to read all available data, so in order to read a 64MB block it allocates a 128MB buffer.

Some things we can do to relax memory usage:

• Eliminate ioutil.ReadAll. Since we know the anticipated amount of data for both PUT (from the Content-Length header) and GET (from the file size on disk) we can allocate exactly the amount of space we need and use that buffer.
• Possibly: remove the requirement that Keep return a 402 Corruption error on a corrupt block. If Keep is allowed to return the block anyway and simply log an alert if the block is found to be corrupt, GET can allocate a relatively small buffer (e.g. 1MB) for reads.