Arvados

This looks pretty good, just one comment on the algorithm. If the requests are timing out instead of failing fast, you may up waiting significantly longer than $timediff before giving up: a five minute job (300 seconds) would do 8 retries, if there is a 60 second timeout plus backoff that means it would spend well over 10 minutes retrying before giving up. Suggest tweaking retry_op() to try/retry (with backoff) until the wait time is exceeded, and at least three times? Something like:

my $wait = 1;
my $giveup_time = time + calculate_giveup_time();
while (time < $giveup_time) {
    sleep($wait) if $wait > 1;
    $wait *= 2; 
    my $result = eval { $operation->(@_); };
    if (!$@) {
      return $result;
    }
}

Peter Amstutz wrote:

If the requests are timing out instead of failing fast, you may up waiting significantly longer than $timediff before giving up

Yeah, that's an issue. Fixed in 7e35706 and ready for another look. Thanks.