Feature #2960

Keep can stream GET and PUT requests

Added by Tim Pierce almost 7 years ago. Updated over 1 year ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:

Related issues

Related to Arvados - Story #9760: Remove intermediary checksum from Keep store and use cut through block forwardingRejected08/08/2016

Related to Arvados - Feature #10541: [Keep] Share buffers between overlapping/consecutive GET requests for the same blockNew

Blocks Arvados Epics - Story #16516: Run Keepstore on local compute nodesNew07/01/202110/31/2021


#1 Updated by Tim Pierce almost 7 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from Keep streams GET and PUT requests to Keep can stream GET and PUT requests

Current implementation of Keep @ a4378cd handles both GET and PUT requests by reading an entire blob (up to 64MB) into memory, performing checksums and any other policy checks, and then delivering the block appropriately. With many concurrent clients, this can bring Keep memory usage up to 500MB and more.

A more memory-efficient solution, and lower latency from the client's point of view, would be to stream data between the client and disk as much as possible with smaller buffers. The keepclient package has some useful tools for this.

Some issues to attend to:
  • When --serialize-io is enabled, Keep must continue to read an entire block and buffer it, or else a single slow client may block other clients while they wait for disk access.
  • GET cannot issue a 402 Corruption error without hashing the block first. Possible solutions include:
    • Read the block twice through a small buffer: once to calculate the checksum, and again to stream it to the client.
    • Deliver any corruption status in the second part of a multipart/mixed message.
    • Abandon any attempt to report corruption to the client and simply log it to the data manager, on the grounds that the client is already responsible for checking for corrupt blocks anyway.
  • PUT checking for collisions, which currently relies on GetBlock, will need to be rethought to handle streaming data.

#2 Updated by Tim Pierce almost 7 years ago

  • Category set to Keep

#3 Updated by Ward Vandewege 11 months ago

  • Blocks Story #16516: Run Keepstore on local compute nodes added

Also available in: Atom PDF