Arvados

I'm seeing much faster upload than that with bmon. How are you measuring the arv put speed?

I note that you are doing something like aws s3 get | arv put. How much time is spent waiting for aws s3 get ?

I am transferring a small portion of files located on an Amazon S3 bucket to su92l. I'm using the '/data' directory as a staging area to download ~2Gb portions from the Amazon cluster to the staging area, uploading to Keep, deleting the staging files, then repeating until all files have been uploaded to Keep.

Using Amazon's 'aws cp' tool, I'm seeing around 80Mb/s download. Uploading to Keep using 'arv-put' I was seeing 0.5Mb/s.

Feel free to look at the source '/data/abram/dbgap.pgp.xfer.20150707/transfer_from_s3_to_su92l.sh'.

As of this writing, I'm roughly measuring upload to Keep at ~10Mb/s.

As of this writing it's slowed down again and no longer getting ~10Mb/s.

A sampling of upload times:

2351M 2m11.719s
2265M 2m6.404s
2361M 19m46.000s
2307M 1m35.493s
5202M 4m47.064s
2354M 4m41.263s
2278M 29m4.064s
2384M 2m41.059s
2603M 27m31.338s
2407M 1m21.680s
2593M 1m35.253s

...

2639M 35m21.532s