Arvados

A typical use case is a CWL workflow that downloads files from a remote server, and scatters over a list of URLs to do so. The remote site does not have infinite bandwidth; the workflow author should be able to specify e.g. downloading no more than 10 URLs concurrently so as not to overwhelm the other side, while scattering over a list of thousands URLs.

Another use case for this would be site-wide: an admin would like to restrict the number of compute resources that can be consumed by a single workflow at any one time. This would help with scheduling fairness.

• (use case 1) CWL author/submitter can limit the number of concurrently running jobs in a scatter step (via a hint?)
• (use case 2) Global config flag to restrict number of concurrent jobs for one workflow.