Project

General

Profile

Actions

Feature #18163

open

[Crunch] add CWL extension to limit the number of concurrently running jobs during a scatter

Added by Ward Vandewege 10 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

A typical use case is a CWL workflow that downloads files from a remote server, and scatters over a list of URLs to do so. The remote site does not have infinite bandwidth; the workflow author should be able to specify e.g. downloading no more than 10 URLs concurrently so as not to overwhelm the other side, while scattering over a list of thousands URLs.

Another use case for this would be site-wide: an admin would like to restrict the number of compute resources that can be consumed by a single workflow at any one time. This would help with scheduling fairness.

Possible implementation:
  • (use case 1) CWL author/submitter can limit the number of concurrently running jobs in a scatter step (via a hint?)
  • (use case 2) Global config flag to restrict number of concurrent jobs for one workflow.

Related issues

Related to Arvados Epics - Story #17848: Improve a-c-r usabilityIn Progress07/01/202108/31/2022

Actions
Actions #1

Updated by Ward Vandewege 10 months ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege 6 months ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege 6 months ago

  • Description updated (diff)
Actions #4

Updated by Ward Vandewege 5 months ago

Actions

Also available in: Atom PDF