Arvados

Node manager currently shuts down nodes based on three criteria: is the node idle, is the queue empty, and is the shutdown window open.

Unfortunately, this doesn't take into account recent activity. It is a common development pattern to run a job, make some tweaks, and then re-submit it. It is also common for pipeline runners to have gap in time between submitting jobs as it looks at the output of a previous job and builds the next job.

Node manager should have a configurable option to set a minimum time the node must be idle before shutting it down, to address these kinds of race conditions where a node gets (inconveniently) shut down the middle of actual work.