Arvados

From #Arvados I got a pointed to a potential duplication issue in older arv-put as: https://dev.arvados.org/issues/6358

Report from a typical Keepstore server for 3 typical Keepstore filetrees:

manager@keepstore07:~$ sudo du -sm /var/lib/keepstore/gcam1-keep-4[345]
1047961 /var/lib/keepstore/gcam1-keep-43
1047976 /var/lib/keepstore/gcam1-keep-44
1047960 /var/lib/keepstore/gcam1-keep-45

manager@keepstore07:~$ df -T -BG /var/lib/keepstore/gcam1-keep-43 /var/lib/keepstore/gcam1-keep-44 /var/lib/keepstore/gcam1-keep-45
Filesystem     Type 1G-blocks  Used Available Use% Mounted on
/dev/vdc1      xfs      1024G 1024G        1G 100% /var/lib/keepstore/gcam1-keep-43
/dev/vdd1      xfs      1024G 1024G        1G 100% /var/lib/keepstore/gcam1-keep-44
/dev/vde1      xfs      1024G 1024G        1G 100% /var/lib/keepstore/gcam1-keep-45

manager@keepstore07:~$ df -i /var/lib/keepstore/gcam1-keep-43 /var/lib/keepstore/gcam1-keep-44 /var/lib/keepstore/gcam1-keep-45
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/vdc1       35982 20475 15507   57% /var/lib/keepstore/gcam1-keep-43
/dev/vdd1       28438 20469  7969   72% /var/lib/keepstore/gcam1-keep-44
/dev/vde1       36414 20475 15939   57% /var/lib/keepstore/gcam1-keep-45

Data Manager report appended. The 447178 unattached blocks it reports match fairly well the size of the 25GiB collection that was reported as 100% uploaded. What is perplexing is that re-uploading the very same files goes as far as 30% as per above reports and then results in a no-space message.

2016/03/23 17:00:35 Returned 10 keep disks
2016/03/23 17:00:35 Replication level distribution: map[1:936463 2:25380 3:3]
2016/03/23 17:00:38 Blocks In Collections: 514668, 
Blocks In Keep: 961846.
2016/03/23 17:00:38 Replication Block Counts:
 Missing From Keep: 0, 
 Under Replicated: 0, 
 Over Replicated: 22639, 
 Replicated Just Right: 492029, 
 Not In Any Collection: 447178. 
Replication Collection Counts:
 Missing From Keep: 0, 
 Under Replicated: 0, 
 Over Replicated: 22, 
 Replicated Just Right: 395.
2016/03/23 17:00:38 Blocks Histogram:
2016/03/23 17:00:38 {Requested:0 Actual:1}:     444434
2016/03/23 17:00:38 {Requested:0 Actual:2}:       2744
2016/03/23 17:00:38 {Requested:1 Actual:1}:     492029
2016/03/23 17:00:38 {Requested:1 Actual:2}:      22636
2016/03/23 17:00:38 {Requested:1 Actual:3}:          3

So thanks to #6358 I discovered (or rediscovered) ARVADOS_DEBUG=2 and applied to a re-upload it showed that:

• arv-put picks "some" Keepstore (probably according to the sorting order mentioned in #6358).
• Tries to PUT the block to it, regardless of whether it may or may not be already on it. and if the PUT fails, goes on to the next Keepstore.
• Eventually it will find a Keepstore where the PUT does not fail, and if it by chance it is the one where the block is already stored it won't be stored twice, else it will be.

This is rather disappointing, as it means that failed uploads create usually unwanted duplicates, that is uploads are not idempotent as to "pragmatics".

Also there is the terrible problem that if an upload of say 25TB lasts many days and perchance the Data Manager runs before the collection's manifest is registered in the API there might be some big disappointment. IIRC this is an aspect that is being worked on, I guess with a black-grey-white state system.

For arv-put these might be possible improvements (several hopefully non-critical details omitted, like replication):

• The Data Managers or the Keepstores maintain in the API server database periodically updated list of all blocks present but not registered in any collection manifest.
• Then arv-put optionally checks existing manifests and that list. If there is no list or the "some" blocks are not present in it, they get uploaded to "some" Keepstore.
• If there is a list and "some" blocks are present in it:
  • arv-put sends a PUT request listing those hashes to the '/reserve' endpoint of all Keepstores.
  • Keepstores reply with a status per-hash: 0="have-and-reserved", 1=dont-have-and-waiting, 2=dont-know.
  • Hashes for which all statuses are "dont-have-and-waiting" or "dont-know": PUT the hash and block to some Keepstore's '/upload' endpoint.
  • At the end when hash is registered in a manifest sent to the API server, send a PUT to endpoint '/registered' of the relevant Keepstore.
  • A Keepstore will refuse to delete a block between its hash being PUT to the '/reserve' endpoint and it being listed in a PUT to the '/registered' endpoint.

IIRC in a previous discussion someone mentioned a more persistent mechanism for "grey" status (uploaded but not yet registered), like uploading or hard-linking the block in a directory like 'incoming' on the Keepstore volume.

To discuss more on #Arvados I guess.

Put another way, the current algorithm for selecting a Keepstore as the destination for a block results in no duplication among Keepstores only if the number of Keepstores never changes and (maybe) have the same amount of free space.

Just noted on IRC that at one point we had all keepstores 100% full, and 100% of blocks being uploaded having already been uploaded. In that case I would have expected all re-uploads to succeed, but they all failed instead instead about 30% in.

We covered this on IRC, but to summarize:

Currently, for each directory arv-put concatenates all the files into a single "stream", and it is the "stream" that is chunked into 64 MiB blocks rather than individual files. This means files are not guaranteed to fall on block boundaries and can start in the middle of a block immediately following the end of a previous file. As a result, if files are uploaded in a different order, this results in a different "stream" which is likely to yield different blocks.

This is not an inherit property of Keep. Data chunking decisions are made at the client level during upload, so we can change the data chunking policy without changing any of the keep infrastructure. For example, creating collections with writable arv-mount ("arv-mount --read-write") creates a separate block stream for each file.

However, this obviously makes deduplication less effective, so I've filed #8791 to change this behavior.