Story #3491

[Keep] Support transparent compression of blocks in Keep

Added by Peter Amstutz over 4 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Support automatic compression of blocks in the Keep server. Proposed design:

  • Keep server can accept PUT blocks compressed with gzip, or use gzip to compress uncompressed blocks before saving to disk. Compare compressed/uncompressed sizes to ensure that compression isn't adding unnecessary overhead.
  • On GET, Keep clients provide the "Accept-Encoding: gzip" header, and the server responds with "Content-Encoding: gzip" and spools the compressed data directly off disk.
  • Keep client decompresses the data before delivering it to the application.
Benefits:
  • Support random access into large files without needing special file formats or explicit application support.
  • Reduce disk and network usage across the board.
  • Transparent to user
Drawbacks:
  • Adds a "decompress and then re-compress on Keep block boundaries" step when working with a collection that's already compressed at the file level.
  • May increase latency and client overhead because each block needs to be decompressed in order to use it

History

#1 Updated by Peter Amstutz over 4 years ago

  • Subject changed from Support transparent compression of blocks in Keep to [Keep] Support transparent compression of blocks in Keep
  • Description updated (diff)
  • Category set to Keep

#2 Updated by Tom Clegg over 4 years ago

  • Target version set to Deferred

Also available in: Atom PDF