Project

General

Profile

Feature #18163

Updated by Ward Vandewege almost 3 years ago

A typical use case is a CWL workflow that downloads files from a remote server, and scatters over a list of URLs to do so. The remote site does not have infinite bandwidth; the workflow author should be able to specify e.g. downloading no more than 10 URLs concurrently so as not to overwhelm the other side, while scattering over a list of thousands URLs. 

 Another use case for this would be site-wide: an admin would like to restrict the number of compute resources that can be consumed by a single workflow at any one time. This would help with scheduling fairness. 

 Possible implementation: 
 * (use case 1) CWL author/submitter can limit the number of concurrently running jobs in a scatter step (via a hint?) 
 * (use case 2) Global config flag to restrict number of concurrent jobs for one workflow.

Back