Arvados

Background:

Currently, nodes will spin up if the job is ready to be run, or in queue. In template talk, a job will start running if all the requirements of the job is met. Specifically, I am looking at the "output_of" parameter. This is usually the basis for whether my jobs will begin to run or not. The current node manager model works for if pipelines are linear, i.e. A > B > C > D ... The amount of nodes here should stay the same, if A needs 5 nodes, B needs 5 nodes, C needs 5, etc. Then, node manager will spawn 5 nodes for A and then once B starts, the same 5 nodes will be used.

Problem:

What if pipelines are non-linear? I.e. A > (B && C && D && E)? If we assume all jobs need 5 nodes, then A will spawn 5 nodes, then B, C, D, and E will all spawn at the same time because A has finished. Therefore, B, C, D, E will each cause node manager to spawn 5 nodes, thus, creating bloating the total node count to 20. But, what if these jobs are quick, on the order of minutes? We don't need to spawn 20 nodes to maximize speed. We would like to cut down on costs by only using the same 5 nodes we were on. The limitation we currently have is that there is no way to keep using the same 5 nodes we are on. We are limited by the "output_of" parameter.

Proposed Solution(s):

A way to stay on the same nodes that the pipeline is currently using. (Would fix docker image installs onto the node when a new node spins up)

A pipeline template switch that says, don't spawn new nodes, just use ones that are currently being used in the pipeline or use idle nodes.