Arvados

Going a bit further, we could also implement a rolling hash function to choose block boundaries during file upload, to improve deduplication even more:

https://en.wikipedia.org/wiki/Rolling_hash

https://crypto.stackexchange.com/questions/16082/cryptographically-secure-keyed-rolling-hash-function

I have written a related feature request with a different motivation (and I know some of you don't think it is important...) than deduplication here, arguing that per-collection or per-file "streams" should be an option to arv-put:

https://dev.arvados.org/issues/8992

Peter Grandi wrote:

I have written a related feature request with a different motivation (and I know some of you don't think it is important...) than deduplication here, arguing that per-collection or per-file "streams" should be an option to arv-put:

https://dev.arvados.org/issues/8992

I understand the current story description is technical enough that it's not easy to tell, but what this ticket proposes is to have the default behavior be at least close to the per-file behavior you request in #8992. We might continue to combine "small" files into a single block (where the definition of "small" is TBD, and may be configurable, but the default threshold would probably be between 4 and 16 MiB), but we would make sure larger files get stored as an independent set of blocks, for the deduplication benefits you've talked about there and on IRC.

If that was the default behavior of arv keep put, would that be enough to obviate the need for #8992?

Peter Amstutz wrote:

CollectionWriter should ensure that large files are aligned to block boundaries

Do we actually have to change the behavior of CollectionWriter? Or would it be sufficient to change the behavior of arv keep put—either by having it "flushing" CollectionWriter more often, or upgrading it to use the Collection class instead?

I'd rather just change arv keep put, and not muck with CollectionWriter. If there's some reason that's not good enough from a user perspective, it'd be good to know.