Bug #20257

Updated by Peter Amstutz about 1 year ago

User reported that, using the http download feature of arvados-cwl-runner, there is no timeout for stuck downloads (where the connection hasn't been dropped but no progress seems to be made). 

 Possible solutions: 

 * Calculate a timeout based on file size and minimum bandwidth and use that with the existing Requests code 
 ** the only problem is we probably have to use a HEAD call to get the content length, and sometimes (such as S3 signed URLs) you can't use HEAD 
 * Switch the download code to use pyCURL and use pycurl.LOW_SPEED_TIME / pycurl.LOW_SPEED_LIMIT options to determine when to abandon poorly performing downloads 

 Sort of related, it seems to be doing integer division to calculate MiB/s, I only see 0.00, 1.00 or 2.00 and not fractional download rates that I would expect.