Project

General

Profile

Actions

Feature #5712

closed

Smarter node allocation for small jobs

Added by Bryan Cosca almost 9 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Story points:
-

Description

Background:

Currently, nodes will spin up if the job is ready to be run, or in queue. In template talk, a job will start running if all the requirements of the job is met. Specifically, I am looking at the "output_of" parameter. This is usually the basis for whether my jobs will begin to run or not. The current node manager model works for if pipelines are linear, i.e. A > B > C > D ... The amount of nodes here should stay the same, if A needs 5 nodes, B needs 5 nodes, C needs 5, etc. Then, node manager will spawn 5 nodes for A and then once B starts, the same 5 nodes will be used.

Problem:

What if pipelines are non-linear? I.e. A > (B && C && D && E)? If we assume all jobs need 5 nodes, then A will spawn 5 nodes, then B, C, D, and E will all spawn at the same time because A has finished. Therefore, B, C, D, E will each cause node manager to spawn 5 nodes, thus, creating bloating the total node count to 20. But, what if these jobs are quick, on the order of minutes? We don't need to spawn 20 nodes to maximize speed. We would like to cut down on costs by only using the same 5 nodes we were on. The limitation we currently have is that there is no way to keep using the same 5 nodes we are on. We are limited by the "output_of" parameter.

Proposed Solution(s):

A way to stay on the same nodes that the pipeline is currently using. (Would fix docker image installs onto the node when a new node spins up)

A pipeline template switch that says, don't spawn new nodes, just use ones that are currently being used in the pipeline or use idle nodes.


Related issues

Related to Arvados - Idea #4299: [Node Manager] Support a configurable throttle for creating cloud nodesRejected10/23/2014Actions
Related to Arvados - Feature #5768: [Node Manager] Add max_node_per_user configNewActions
Actions #1

Updated by Brett Smith almost 9 years ago

#4299 would go a long way to address this, and support other Node Manager features, without requiring the user to specify anything.

Actions #2

Updated by Tom Clegg almost 9 years ago

  • Target version deleted (Bug Triage)
Actions #3

Updated by Peter Amstutz about 4 years ago

  • Status changed from New to Closed
Actions

Also available in: Atom PDF