Arvados

Testing Keepstore. Had keep0 with a 100GB volume. Created 2 new filesystem, 03 and 04, and copied the contents of the existing one into 03 and delete the existing one, and restarted the daemon. Created keep1 with filesystems 01 and 02. All 4 filesystems are 1TiB. Registered keep1 with the API server.

When uploading with arv-put a small number of files each of a few GB plus 1 file of 60GB the 64MiB blobs get distributed as follows:

keep0:

$ find /var/lib/keepstore/gcam1-keep-04 -type f | wc -l
2264
$ find /var/lib/keepstore/gcam1-keep-03 -type f | wc -l
3268

keep1:

$ find /var/lib/keepstore/gcam1-keep-02 -type f | wc -l
3
$ find /var/lib/keepstore/gcam1-keep-01 -type f | wc -l                                                                             
2

That seems very strange to me. I can understand the filesystem 03 has more blobs then 04 because it has the "old" blobs.

Looking at a couple of the 5 blobs on keep1 in 01 and 02 they seem to belong to files stored almost entirely on keep0. What seems strange to me is both that:

• Files are not evenly distributed between keep0 and keep1.
• They are evenly distributed between 03 and 04 on keep0 but some stray blobs end up (apparently evenly distributed) on keep1.