Idea #3491
Updated by Peter Amstutz over 9 years ago
Support automatic compression of blocks in the Keep server. Proposed design:
* Keep server can accept PUT blocks compressed with gzip, or use gzip to compress uncompressed blocks before saving to disk. Compare compressed/uncompressed sizes to ensure that compression isn't adding unnecessary overhead.
* On GET, Keep clients provide the "Accept-Encoding: gzip" header, and the server responds with "Content-Encoding: gzip" and spools the compressed data directly off disk.
* Keep client decompresses the data before delivering it to the application.
Benefits:
* Support random access into large files without needing special file formats or explicit application support.
* Reduce disk and network usage across the board.
* Transparent to user
Drawbacks:
* Adds a "decompress and then re-compress on Keep block boundaries" step when working with a collection that's already compressed at the file level.
* May increase latency and client overhead because each block needs to be decompressed in order to use it