Arvados

I tried uploading an arvados git repo checkout (3646 files in 979 dirs) to tordo from a shell node, using a one-liner with curl and xargs.

Admittedly the testing plan adds a lot of overhead, which would tend to mask performance issues. But the new version (with repacking) finishes faster than the old version. So it feels pretty safe to say we haven't introduced any significant performance problems.

time find arvados -type d | while read f; do echo curl -f -X MKCOL -H "'Authorization: Bearer $ARVADOS_API_TOKEN'" "'https://download.tordo.arvadosapi.com/c=$uuid/${f#arvados/}'"; done | xargs -n1 -d \\n bash -c
time find arvados -type f | while read f; do echo curl -f -X PUT -H "'Authorization: Bearer $ARVADOS_API_TOKEN'" -H "'Content-Type: application/octet-stream'" -T "'$f'" "'https://download.tordo.arvadosapi.com/c=$uuid/${f#arvados/}'" "|| echo $f"; done | xargs -n1 -d \\n -P6 bash -c

Tom Clegg wrote in #note-4:

I tried uploading an arvados git repo checkout (3646 files in 979 dirs) to tordo from a shell node, using a one-liner with curl and xargs. … it feels pretty safe to say we haven't introduced any significant performance problems.

Agreed, lgtm, thanks.