Project

General

Profile

Actions

Bug #13166

closed

[node manager] wishlist should consist of top priority containers

Added by Peter Amstutz about 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Description

Node manager calculates a "wishlist" which is how many nodes of each size are desired.

The current behavior reports the entire queue, the creates nodes starting with the largest size and working its way down to the smallest. The rationale being that small jobs can run on large nodes but not the other way around.

However, #12199 changes the behavior such that containers will be scheduled on specific node sizes. If the queue is much larger than the maximum number of nodes, this could lead to a situation in which the top of the queue consists of small jobs, but only large nodes are available. In this case, it will either schedule jobs out of order (with low-priority large-node jobs jumping the queue) or deadlock.

To fix:

  • After getting the contents of squeue, sort in decending order by slurm priority
  • Only consider the top (max_nodes - up_nodes) items in the wishlist and discard the remaining

Subtasks 1 (0 open1 closed)

Task #13273: Review 13166-nodemanager-whishlistClosedPeter Amstutz03/26/2018Actions

Related issues

Related to Arvados - Idea #12552: When priority is equal, the children of an earlier-submitted workflow should run firstResolvedTom Clegg11/03/2017Actions
Related to Arvados - Bug #12199: Don't schedule jobs on nodes which are too much bigger than requestedResolvedTom Clegg01/29/2018Actions
Actions #1

Updated by Peter Amstutz about 6 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz about 6 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Clegg about 6 years ago

  • Related to Idea #12552: When priority is equal, the children of an earlier-submitted workflow should run first added
Actions #4

Updated by Peter Amstutz about 6 years ago

  • Related to Bug #12199: Don't schedule jobs on nodes which are too much bigger than requested added
Actions #5

Updated by Peter Amstutz about 6 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz about 6 years ago

  • Status changed from In Progress to New
Actions #7

Updated by Tom Morris about 6 years ago

  • Assigned To set to Peter Amstutz
  • Target version changed from To Be Groomed to 2018-03-28 Sprint
Actions #8

Updated by Lucas Di Pentima about 6 years ago

  • Assigned To changed from Peter Amstutz to Lucas Di Pentima
Actions #9

Updated by Lucas Di Pentima about 6 years ago

  • Status changed from New to In Progress
Actions #10

Updated by Lucas Di Pentima about 6 years ago

Updates at 93d9c63e1fd4d4c268aa9050406b9a212f3d4a22 - branch 13166-nodemanager-whishlist

Changed wishlist behavior as requested, all tests ran OK on 2dd214985e9727bf0b5fbf11b0e39c3e7d3cc5c4 but they're failing on Jenkins (https://ci.curoverse.com/job/developer-run-tests-services-nodemanager/164/console) but running OK on my VM.

Actions #11

Updated by Peter Amstutz about 6 years ago

Comment is missing the addition of %Q for priority.

# cpus, memory, tempory disk space, reason, job name, feature constraints
squeue_out = subprocess.check_output(["squeue", "--state=PENDING", "--noheader", "--format=%c|%m|%d|%r|%j|%f|%Q"])

An in-place sort would be slightly more efficient:

queuelist.sort(key=lambda x: x.get('priority', 1), reverse=True)

In order to test the wishlist ordering, you should have a variation of this test where the "big" node is at the end of the list and check that it only boots small nodes.

    def test_node_max_nodes_two_sizes(self):
...
        self.make_daemon(want_sizes=[small, small, big, small],
                         avail_sizes=avail_sizes, max_nodes=3)

I'm not quite sure what is going on with the integration test. The test framework does have a failure mode where it crashes with "IOError: [Errno 11] Resource temporarily unavailable" I haven't gotten to the bottom of what that happens.

Actions #12

Updated by Peter Amstutz about 6 years ago

I think this fixes the integration tests:

diff --git a/services/nodemanager/tests/integration_test.py b/services/nodemanager/tests/integration_test.py
index 48fc3951e..508e62663 100755
--- a/services/nodemanager/tests/integration_test.py
+++ b/services/nodemanager/tests/integration_test.py
@@ -342,7 +342,6 @@ def main():
                 (r".*Cloud node (\S+) is now paired with Arvados node (\S+) with hostname (\S+)", node_paired),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Suggesting shutdown because node state is \('idle', 'open', .*\)", remaining_jobs),
-                (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown)
             ],
@@ -369,7 +368,6 @@ def main():
                 (r".*Cloud node (\S+) is now paired with Arvados node (\S+) with hostname (\S+)", node_paired),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Suggesting shutdown because node state is \('idle', 'open', .*\)", remaining_jobs),
-                (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*sending request", jobs_req),


Actions #13

Updated by Lucas Di Pentima about 6 years ago

Updates at b17321254
Test run: https://ci.curoverse.com/job/developer-run-tests/667/

  • Updated integration tests
  • Updated squeue call comment & list ordering
  • Added test to check wishlist ordering
Actions #14

Updated by Peter Amstutz about 6 years ago

Lucas Di Pentima wrote:

Updates at b17321254
Test run: https://ci.curoverse.com/job/developer-run-tests/667/

  • Updated integration tests
  • Updated squeue call comment & list ordering
  • Added test to check wishlist ordering

This LGTM, thanks.

Actions #15

Updated by Lucas Di Pentima about 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #16

Updated by Tom Morris over 5 years ago

  • Release set to 17
Actions

Also available in: Atom PDF