Arvados

What I ran into today:

1) create an ephemeral arvados cluster with our kubernetes prototype
2) arv-put a file to it from my local system. Arv-put creates a local cache directory in .cache/arvados/arv-put/
3) blow away the arvados cluster, and re-create it
4) re-run the arv-put command from my local system:

$ arv-put 4xphq-8i9sb-fmwod1qn74cemdp.log.txt 
2019-02-27 22:05:07 arvados.arv_put[103627] INFO: Resuming upload from cache file /home/wvandewege/.cache/arvados/arv-put/0a16eb3c98df444230504d71edb7d7c1
0M / 0M 100.0% 2019-02-27 22:05:07 arvados.arv_put[103627] INFO: 

2019-02-27 22:05:07 arvados.arv_put[103627] INFO: Collection saved as 'Saved at 2019-02-27 22:05:07 UTC by wvandewege@shell.qr1hi.arvadosapi.com'
vwxyz-4zz18-v5p53u2m8rdl17i

Arv-put happily creates the collection without checking the blocks. The blocks don't exist.

Workaround: blow away cache directory and re-run the arv-put command.

While this scenario is unusual today, we are going to a future with more ephemeral arvados installations, and this will become a more common occurrence.

Proposed solution: arv-put should double check that (some of? all of?) the blocks exist before creating a collection from a local cache.