Feature #8551

[Node Manager] Minimum idle time before node is eligible for shutdown

Added by Peter Amstutz about 3 years ago. Updated about 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Node Manager
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Node manager currently shuts down nodes based on three criteria: is the node idle, is the queue empty, and is the shutdown window open.

Unfortunately, this doesn't take into account recent activity. It is a common development pattern to run a job, make some tweaks, and then re-submit it. It is also common for pipeline runners to have gap in time between submitting jobs as it looks at the output of a previous job and builds the next job.

Node manager should have a configurable option to set a minimum time the node must be idle before shutting it down, to address these kinds of race conditions where a node gets (inconveniently) shut down the middle of actual work.

History

#1 Updated by Peter Amstutz about 3 years ago

  • Description updated (diff)

#2 Updated by Peter Amstutz about 3 years ago

  • Category set to Node Manager

#4 Updated by Brett Smith about 3 years ago

  • Target version set to Arvados Future Sprints

Also available in: Atom PDF