Feature #16519

[keepstore] optimize md5sum calculations

Added by Ward Vandewege over 1 year ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

There is now a Go package to speed up md5sum calculations when the hardware supports it (AVX/AVX2 extensions, which are common):

https://github.com/minio/md5-simd

which is described here:

https://blog.min.io/accelerating-aggregate-md5-hashing-up-to-800-with-avx512-2/

Keepstore should leverage this library to speed up its hashing, if the hardware it runs on supports the necessary extensions.

Ideally, this goes into our codebase in a such a way that all our Go code that calculates md5sums leverages it automatically.


Related issues

Related to Arvados Epics - Story #16516: Run Keepstore on local compute nodesIn Progress10/01/202111/30/2021

Related to Arvados - Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstoreNew

Related to Arvados - Feature #16513: Get reference Keep performance numbers for Keep-on-S3Resolved06/15/2020

History

#1 Updated by Ward Vandewege over 1 year ago

  • Related to Story #16516: Run Keepstore on local compute nodes added

#2 Updated by Ward Vandewege over 1 year ago

  • Description updated (diff)

#3 Updated by Ward Vandewege over 1 year ago

  • Description updated (diff)

#4 Updated by Ward Vandewege over 1 year ago

  • Related to Feature #16518: [keep] Allow clients to set a header to disable md5sum calculations in keepstore added

#5 Updated by Ward Vandewege over 1 year ago

  • Description updated (diff)

#6 Updated by Ward Vandewege over 1 year ago

  • Related to Feature #16513: Get reference Keep performance numbers for Keep-on-S3 added

Also available in: Atom PDF