Bug #6510
closed
Keep upload using 'arv-put' from 'abe.shell.su92l' is getting 0.5Mb/s upload rate
Description
Uploading a collection from my shell node, abe.shell.su92l, using 'arv-put' is getting a rate of 0.5Mb/s. I'm trying to upload ~150 2Gb directories which will take around 7 days at this rate. I would expect this upload to take a few hours.
Updated by Ward Vandewege almost 9 years ago
I'm seeing much faster upload than that with bmon. How are you measuring the arv put speed?
I note that you are doing something like aws s3 get | arv put. How much time is spent waiting for aws s3 get ?
Updated by Abram Connelly almost 9 years ago
I am transferring a small portion of files located on an Amazon S3 bucket to su92l. I'm using the '/data' directory as a staging area to download ~2Gb portions from the Amazon cluster to the staging area, uploading to Keep, deleting the staging files, then repeating until all files have been uploaded to Keep.
Using Amazon's 'aws cp' tool, I'm seeing around 80Mb/s download. Uploading to Keep using 'arv-put' I was seeing 0.5Mb/s.
Feel free to look at the source '/data/abram/dbgap.pgp.xfer.20150707/transfer_from_s3_to_su92l.sh'.
As of this writing, I'm roughly measuring upload to Keep at ~10Mb/s.
Updated by Abram Connelly almost 9 years ago
As of this writing it's slowed down again and no longer getting ~10Mb/s.
Updated by Abram Connelly almost 9 years ago
A sampling of upload times:
2351M 2m11.719s
2265M 2m6.404s
2361M 19m46.000s
2307M 1m35.493s
5202M 4m47.064s
2354M 4m41.263s
2278M 29m4.064s
2384M 2m41.059s
2603M 27m31.338s
2407M 1m21.680s
2593M 1m35.253s
...
2639M 35m21.532s
Updated by Tom Morris over 7 years ago
- Assigned To set to Ward Vandewege
Assigning to Ward for triage. Is there anything useful we can extract from this bug report or should it be closed as too ancient?
Updated by Ward Vandewege almost 5 years ago
- Status changed from New to Resolved