Actions
Feature #16519
open
[keepstore] optimize md5sum calculations
Story points:
-
Release:
Release relationship:
Auto
Description
There is now a Go package to speed up md5sum calculations when the hardware supports it (AVX/AVX2 extensions, which are common):
https://github.com/minio/md5-simdwhich is described here:
https://blog.min.io/accelerating-aggregate-md5-hashing-up-to-800-with-avx512-2/Keepstore should leverage this library to speed up its hashing, if the hardware it runs on supports the necessary extensions.
Ideally, this goes into our codebase in a such a way that all our Go code that calculates md5sums leverages it automatically.
Related issues
Updated by Ward Vandewege over 4 years ago
- Related to Idea #16516: Run Keepstore on local compute nodes added
Updated by Ward Vandewege over 4 years ago
- Related to Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore added
Updated by Ward Vandewege over 4 years ago
- Related to Feature #16513: Get reference Keep performance numbers for Keep-on-S3 added
Updated by Peter Amstutz almost 3 years ago
- Related to Idea #18342: Keep performance optimization added
Updated by Peter Amstutz almost 3 years ago
- Related to deleted (Idea #16516: Run Keepstore on local compute nodes)
Actions