Task #2954

Bug #2865: Reduce Keep server memory use

Rewrite Get and Put to minimize unnecessary allocation

Added by Tim Pierce over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tim Pierce
Category:
-
Start date:
05/26/2014
Due date:
% Done:

0%

Estimated time:
8.00 h

Description

Both volume.Get and main.ReadAtMost use ioutil.ReadAll to read all available data. As Peter observed, ReadAll grows a buffer exponentially to read all available data, so in order to read a 64MB block it allocates a 128MB buffer.

Some things we can do to relax memory usage:

  • Eliminate ioutil.ReadAll. Since we know the anticipated amount of data for both PUT (from the Content-Length header) and GET (from the file size on disk) we can allocate exactly the amount of space we need and use that buffer.
  • Possibly: remove the requirement that Keep return a 402 Corruption error on a corrupt block. If Keep is allowed to return the block anyway and simply log an alert if the block is found to be corrupt, GET can allocate a relatively small buffer (e.g. 1MB) for reads.

History

#1 Updated by Tim Pierce over 6 years ago

  • Status changed from New to In Progress

#2 Updated by Tim Pierce over 6 years ago

  • Status changed from In Progress to Resolved
  • Remaining (hours) changed from 8.0 to 0.0

Also available in: Atom PDF