Feature #18163
Updated by Ward Vandewege over 2 years ago
A typical use case is a CWL workflow that downloads files from a remote server, and scatters over a list of URLs to do so. The remote site does not have infinite bandwidth; the workflow author should be able to specify e.g. downloading no more than 10 URLs concurrently so as not to overwhelm the other side, while scattering over a list of thousands URLs.
Another use case for this would be site-wide: an admin would like to restrict the number of compute resources that can be consumed by a single workflow at any one time.
Possible implementation:
* (use case 1) CWL author/submitter can limit the number of concurrently running jobs in a scatter step (via a hint?)
* (use case 2) Global config flag to restrict number of concurrent jobs for one workflow.