Bug #4309

Updated by Peter Amstutz almost 5 years ago

Originally https://arvados.org/issues/3699#note-43

I'm copying a pipeline with a 5990M collection. I noticed this code:

data = src_keep.get(word)
dst_locator = dst_keep.put(data)

See attached image, there's a very clear falloff between blocks -- doing this sequentially isn't optimal. Download and upload could proceed concurrently. Also, it's possible I suspect we could get better utilization if we transferred multiple downloaded 2 blocks at a time (e.g. 2x down / 2x up) by talking to multiple Keep servers. time.