Bug #4309

[SDK] arv-copy collection copy performance

Added by Peter Amstutz about 5 years ago. Updated about 5 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
2.0

Description

Originally https://arvados.org/issues/3699#note-43

I'm copying a pipeline with a 5990M collection. I noticed this code:

                    data = src_keep.get(word)
                    dst_locator = dst_keep.put(data)

See attached image, there's a very clear falloff between blocks -- doing this sequentially isn't optimal. Download and upload could proceed concurrently. Also, it's possible we could get better utilization if we transferred multiple blocks at a time (e.g. 2x down / 2x up) by talking to multiple Keep servers. Consider a producer-consumer pattern using Python queues.

arv-copy-perf.png (25.8 KB) arv-copy-perf.png Peter Amstutz, 10/24/2014 05:22 PM

History

#1 Updated by Peter Amstutz about 5 years ago

#2 Updated by Peter Amstutz about 5 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz about 5 years ago

  • Description updated (diff)

#4 Updated by Peter Amstutz about 5 years ago

  • Subject changed from [SDK] arv-copy performance to [SDK] arv-copy collection copy performance
  • Description updated (diff)

#5 Updated by Radhika Chippada about 5 years ago

  • Target version set to Bug Triage

#6 Updated by Tom Clegg about 5 years ago

  • Target version changed from Bug Triage to Arvados Future Sprints

#7 Updated by Ward Vandewege about 5 years ago

  • Story points set to 2.0

Also available in: Atom PDF