Arvados

I used dev-copy and it was being taxed for memory. I got the following spew of error messages. Nico fixed the issue for today - but we discussed that is would be better to handle this issue more gracefully.

sarah@shell:~/.config/arvados$ arv-copy --src qr1hi --dst dst_clust --dst-git-repo $USER/tutorial qr1hi-p5p6p-9pkaxt6qjnkxhhu
8fce8d374fdb7e780c0017a33ce57398+128: 6M / 6M 100.0%
d341a6f1db391a780d694e240e95e475+3805: 64M / 4799M 1.3% MemoryError: out of memory
Traceback (most recent call last):
File "/usr/local/bin/arv-copy", line 4, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 128, in main
src_arv, dst_arv, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 282, in copy_pipeline_template
pt = copy_collections(pt, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in copy_collections
for v in obj)
File "/usr/lib/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib/python2.7/_abcoll.py", line 547, in update
for key, value in other:
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in <genexpr>
for v in obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in copy_collections
for v in obj)
File "/usr/lib/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib/python2.7/_abcoll.py", line 547, in update
for key, value in other:
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in <genexpr>
for v in obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in copy_collections
for v in obj)
File "/usr/lib/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib/python2.7/_abcoll.py", line 547, in update
for key, value in other:
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in <genexpr>
for v in obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in copy_collections
for v in obj)
File "/usr/lib/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib/python2.7/_abcoll.py", line 547, in update
for key, value in other:
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in <genexpr>
for v in obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in copy_collections
for v in obj)
File "/usr/lib/python2.7/collections.py", line 52, in init
self.__update(*args, **kwds)
File "/usr/lib/python2.7/_abcoll.py", line 547, in update
for key, value in other:
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 334, in <genexpr>
for v in obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 329, in copy_collections
obj = arvados.util.portable_data_hash_pattern.sub(copy_collection_fn, obj)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 319, in copy_collection_fn
dst_col = copy_collection(src_id, src, dst, args)
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/arv_copy.py", line 569, in copy_collection
data = src_keep.get(word)
File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 153, in num_retries_setter
return orig_func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 980, in get
"failed to read {}".format(loc_s), service_errors, label="service")
arvados.errors.KeepReadError: failed to read b0bd8ea685e9f07021166939e73b07c3+67108864+A70ca7e70f8edeaeaacd980e743cbe3c238c66535@5686e18d: service http://keep12.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep15.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep16.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep13.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep14.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep11.qr1hi.arvadosapi.com:25107/ responded with 404 HTTP/1.1 404 Not Found
; service http://keep10.qr1hi.arvadosapi.com:25107/ responded with 0 (23, 'Failed writing body (0 != 16384)')