Bug #20257

Updated by Peter Amstutz over 1 year ago

User reported that, using the http download feature of arvados-cwl-runner, there is no timeout for stuck downloads (where the connection hasn't been dropped but no progress seems to be made). 

 Possible solutions: 

 * Calculate a timeout based on file size and minimum bandwidth and use that with the existing Requests code 
 ** the only problem is we probably have to use a HEAD call to get the content length, and sometimes (such as S3 signed URLs) you can't use HEAD 
 * Switch the download code to use pyCURL and use pycurl.LOW_SPEED_TIME / pycurl.LOW_SPEED_LIMIT options to determine when to abandon poorly performing downloads