Bug #13166
Updated by Peter Amstutz about 6 years ago
Node manager calculates a "wishlist" which is how many nodes of each size are desired. The current behavior reports the entire queue, the creates nodes starting with the largest size and working its way down to the smallest. The rationale being that small jobs can run on large nodes but not the other way around. However, #12199 changes the behavior such that containers will be scheduled on specific node sizes. If the queue is much larger than the maximum number of nodes, this could lead to a situation in which the top of the queue consists of small jobs, but only large nodes are available. In this case, it will either schedule jobs out of order (with low-priority large-node jobs jumping the queue) or deadlock. To fix: * After getting the contents of squeue, sort in decending order by slurm priority * Only consider the top (max_nodes - up_nodes) items in the Start new node sizes based on wishlist and discard the remaining priority order instead of decending node size order