Bug #20257
Updated by Peter Amstutz over 1 year ago
User reported that, using the http download feature of arvados-cwl-runner, there is no timeout for stuck downloads (where the connection hasn't been dropped but no progress seems to be made).
Possible solutions:
* Calculate a timeout based on file size and minimum bandwidth and use that with the existing Requests code
** the only problem is we probably have to use a HEAD call to get the content length, and sometimes (such as S3 signed URLs) you can't use HEAD
* Switch the download code to use pyCURL and use pycurl.LOW_SPEED_TIME / pycurl.LOW_SPEED_LIMIT options to determine when to abandon poorly performing downloads
Sort of related, it seems to be doing integer division to calculate MiB/s, I only see 0.00, 1.00 or 2.00 and not fractional download rates that I would expect.