Actions
Feature #2960
openKeep can stream GET and PUT requests
Start date:
Due date:
% Done:
0%
Estimated time:
Story points:
-
Related issues
Updated by Tim Pierce over 9 years ago
- Tracker changed from Bug to Feature
- Subject changed from Keep streams GET and PUT requests to Keep can stream GET and PUT requests
Current implementation of Keep @ a4378cd handles both GET and PUT requests by reading an entire blob (up to 64MB) into memory, performing checksums and any other policy checks, and then delivering the block appropriately. With many concurrent clients, this can bring Keep memory usage up to 500MB and more.
A more memory-efficient solution, and lower latency from the client's point of view, would be to stream data between the client and disk as much as possible with smaller buffers. The keepclient
package has some useful tools for this.
- When
--serialize-io
is enabled, Keep must continue to read an entire block and buffer it, or else a single slow client may block other clients while they wait for disk access. - GET cannot issue a 402 Corruption error without hashing the block first. Possible solutions include:
- Read the block twice through a small buffer: once to calculate the checksum, and again to stream it to the client.
- Deliver any corruption status in the second part of a multipart/mixed message.
- Abandon any attempt to report corruption to the client and simply log it to the data manager, on the grounds that the client is already responsible for checking for corrupt blocks anyway.
- PUT checking for collisions, which currently relies on
GetBlock
, will need to be rethought to handle streaming data.
Updated by Ward Vandewege over 3 years ago
- Blocks Story #16516: Run Keepstore on local compute nodes added
Updated by Peter Amstutz almost 2 years ago
- Related to Story #18342: Keep performance optimization added
Updated by Peter Amstutz almost 2 years ago
- Blocks deleted (Story #16516: Run Keepstore on local compute nodes)
Updated by Peter Amstutz 2 months ago
- Release deleted (
60) - Target version set to Development 2023-10-25 sprint
Updated by Peter Amstutz 2 months ago
- Target version changed from Development 2023-10-25 sprint to Development 2023-11-08 sprint
Updated by Peter Amstutz 2 months ago
- Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Updated by Peter Amstutz 2 months ago
- Target version changed from Development 2023-11-29 sprint to Development 2023-11-08 sprint
Updated by Peter Amstutz 2 months ago
- Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Updated by Peter Amstutz 2 months ago
- Target version changed from Development 2023-11-29 sprint to Development 2023-11-08 sprint
Updated by Peter Amstutz about 1 month ago
- Target version changed from Development 2023-11-08 sprint to Development 2023-11-29 sprint
Updated by Peter Amstutz about 1 month ago
- Target version changed from Development 2023-11-29 sprint to Development 2024-01-03 sprint
Updated by Peter Amstutz 18 days ago
- Target version changed from Development 2024-01-03 sprint to Development 2024-01-17 sprint
Actions