Story #3491

[Keep] Support transparent compression of blocks in Keep

Added by Peter Amstutz over 5 years ago. Updated over 5 years ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


Support automatic compression of blocks in the Keep server. Proposed design:

  • Keep server can accept PUT blocks compressed with gzip, or use gzip to compress uncompressed blocks before saving to disk. Compare compressed/uncompressed sizes to ensure that compression isn't adding unnecessary overhead.
  • On GET, Keep clients provide the "Accept-Encoding: gzip" header, and the server responds with "Content-Encoding: gzip" and spools the compressed data directly off disk.
  • Keep client decompresses the data before delivering it to the application.
  • Support random access into large files without needing special file formats or explicit application support.
  • Reduce disk and network usage across the board.
  • Transparent to user
  • Adds a "decompress and then re-compress on Keep block boundaries" step when working with a collection that's already compressed at the file level.
  • May increase latency and client overhead because each block needs to be decompressed in order to use it


#1 Updated by Peter Amstutz over 5 years ago

  • Subject changed from Support transparent compression of blocks in Keep to [Keep] Support transparent compression of blocks in Keep
  • Description updated (diff)
  • Category set to Keep

#2 Updated by Tom Clegg over 5 years ago

  • Target version set to Deferred

Also available in: Atom PDF