Project

General

Profile

Actions

Bug #20257

closed

Timeout stuck downloads in arvados-cwl-runner

Added by Peter Amstutz about 1 year ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Story points:
-
Release relationship:
Auto

Description

User reported that, using the http download feature of arvados-cwl-runner, there is no timeout for stuck downloads (where the connection hasn't been dropped but no progress seems to be made).

Possible solutions:

  • Calculate a timeout based on file size and minimum bandwidth and use that with the existing Requests code
    • the only problem is we probably have to use a HEAD call to get the content length, and sometimes (such as S3 signed URLs) you can't use HEAD
  • Switch the download code to use pyCURL and use pycurl.LOW_SPEED_TIME / pycurl.LOW_SPEED_LIMIT options to determine when to abandon poorly performing downloads

Sort of related, it seems to be doing integer division to calculate MiB/s, I only see 0.00, 1.00 or 2.00 and not fractional download rates that I would expect.


Subtasks 1 (0 open1 closed)

Task #20387: Review 20257-http-importResolvedPeter Amstutz04/19/2023Actions

Related issues

Related to Arvados - Idea #20917: Test PySDK with different versions of pycurlNewActions
Actions

Also available in: Atom PDF