Arvados

Bug #4309

Updated by Peter Amstutz over 9 years ago

Originally https://arvados.org/issues/3699#note-43 

 I'm copying a pipeline with a 5990M collection.    I noticed this code: 

 <pre> 
                     data = src_keep.get(word) 
                     dst_locator = dst_keep.put(data) 
 </pre> 

 See attached image, there's a very clear falloff between blocks -- doing this sequentially isn't optimal.    Download and upload could proceed concurrently.    Also, it's possible we could get better utilization if we transferred multiple blocks at a time (e.g. 2x down / 2x up) by talking to multiple Keep servers.    Consider a producer-consumer pattern using "Python queues.":https://docs.python.org/2/library/queue.html

Back