Project

General

Profile

Actions

Feature #18163

open

[Crunch] add CWL extension to limit the number of concurrently running jobs during a scatter

Added by Ward Vandewege over 2 years ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

A typical use case is a CWL workflow that downloads files from a remote server, and scatters over a list of URLs to do so. The remote site does not have infinite bandwidth; the workflow author should be able to specify e.g. downloading no more than 10 URLs concurrently so as not to overwhelm the other side, while scattering over a list of thousands URLs.

Another use case for this would be site-wide: an admin would like to restrict the number of compute resources that can be consumed by a single workflow at any one time. This would help with scheduling fairness.

Possible implementation:
  • (use case 1) CWL author/submitter can limit the number of concurrently running jobs in a scatter step (via a hint?)
  • (use case 2) Global config flag to restrict number of concurrent jobs for one workflow.

Related issues

Related to Arvados Epics - Idea #20273: More CWL runner improvementsNewActions
Actions #1

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Ward Vandewege over 2 years ago

  • Description updated (diff)
Actions #4

Updated by Ward Vandewege about 2 years ago

  • Related to Idea #17848: CWL runner improvements added
Actions #5

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #6

Updated by Peter Amstutz about 1 year ago

  • Related to Idea #20273: More CWL runner improvements added
Actions #7

Updated by Peter Amstutz about 1 year ago

  • Related to deleted (Idea #17848: CWL runner improvements)
Actions #8

Updated by Peter Amstutz about 1 month ago

  • Target version set to Future
Actions

Also available in: Atom PDF