Story #10387

Faster downloading using arv-get

Added by Joshua Randall over 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
10/27/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

As a user wanting to access data kept in keep from an external (non-Arvados) system, I'd like my `arv keep get` requests to be able to fully utilise available network bandwidth for a download from keep. The limiting factor currently seems to be that blocks are downloaded serially starting at the beginning of a manifest and working forward to the end). My expectation would be that destination files could be preallocated and that blocks could then be downloaded concurrently to fill in the files as the blocks arrive. The rate of parallel downloading could be increased until overall throughput stops increasing (indicating the network may be saturated) at which point the client could back off a bit. The maximum number of concurrent connections to each keep server could be limited by configuration.

A request for parallel downloading may be incompatible with options such as `--md5sum` to display the hash of files as they come in (because the hash needs to be computed serially), but as a user I wouldn't mind making the options mutually exclusive and being able to choose either `--parallel` (or whatever) or `--md5sum`.


Related issues

Related to Arvados - Story #7824: [SDKs] arv-get and arv-ls should use new PySDK Collection APIsResolved11/19/2015

History

#1 Updated by Tom Morris over 1 year ago

  • Target version set to Arvados Future Sprints

Also available in: Atom PDF