Project

General

Profile

Actions

Feature #2960

open

Keep can stream GET and PUT requests

Added by Tim Pierce over 9 years ago. Updated 18 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Related issues

Related to Arvados - Story #9760: Remove intermediary checksum from Keep store and use cut through block forwardingRejected08/08/2016

Actions
Related to Arvados - Feature #10541: [Keep] Share buffers between overlapping/consecutive GET requests for the same blockNew

Actions
Related to Arvados Epics - Story #18342: Keep performance optimizationNew08/01/202310/31/2023

Actions
Actions #1

Updated by Tim Pierce over 9 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from Keep streams GET and PUT requests to Keep can stream GET and PUT requests

Current implementation of Keep @ a4378cd handles both GET and PUT requests by reading an entire blob (up to 64MB) into memory, performing checksums and any other policy checks, and then delivering the block appropriately. With many concurrent clients, this can bring Keep memory usage up to 500MB and more.

A more memory-efficient solution, and lower latency from the client's point of view, would be to stream data between the client and disk as much as possible with smaller buffers. The keepclient package has some useful tools for this.

Some issues to attend to:
  • When --serialize-io is enabled, Keep must continue to read an entire block and buffer it, or else a single slow client may block other clients while they wait for disk access.
  • GET cannot issue a 402 Corruption error without hashing the block first. Possible solutions include:
    • Read the block twice through a small buffer: once to calculate the checksum, and again to stream it to the client.
    • Deliver any corruption status in the second part of a multipart/mixed message.
    • Abandon any attempt to report corruption to the client and simply log it to the data manager, on the grounds that the client is already responsible for checking for corrupt blocks anyway.
  • PUT checking for collisions, which currently relies on GetBlock, will need to be rethought to handle streaming data.
Actions #2

Updated by Tim Pierce over 9 years ago

  • Category set to Keep
Actions #3

Updated by Ward Vandewege over 3 years ago

  • Blocks Story #16516: Run Keepstore on local compute nodes added
Actions #4

Updated by Peter Amstutz almost 2 years ago

  • Related to Story #18342: Keep performance optimization added
Actions #5

Updated by Peter Amstutz almost 2 years ago

  • Blocks deleted (Story #16516: Run Keepstore on local compute nodes)
Actions #6

Updated by Peter Amstutz 10 months ago

  • Release set to 60
Actions #7

Updated by Peter Amstutz 2 months ago

  • Release deleted (60)
  • Target version set to Development 2023-10-25 sprint
Actions #8

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Actions #9

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #10

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2023-11-08 sprint
Actions #11

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #12

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2023-11-29 sprint to Development 2023-11-08 sprint
Actions #13

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Actions #14

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Actions #15

Updated by Peter Amstutz 18 days ago

  • Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Actions

Also available in: Atom PDF