Arvados

Updates at 93d9c63e1fd4d4c268aa9050406b9a212f3d4a22 - branch 13166-nodemanager-whishlist

Changed wishlist behavior as requested, all tests ran OK on 2dd214985e9727bf0b5fbf11b0e39c3e7d3cc5c4 but they're failing on Jenkins (https://ci.curoverse.com/job/developer-run-tests-services-nodemanager/164/console) but running OK on my VM.

Comment is missing the addition of %Q for priority.

# cpus, memory, tempory disk space, reason, job name, feature constraints
squeue_out = subprocess.check_output(["squeue", "--state=PENDING", "--noheader", "--format=%c|%m|%d|%r|%j|%f|%Q"])

An in-place sort would be slightly more efficient:

queuelist.sort(key=lambda x: x.get('priority', 1), reverse=True)

In order to test the wishlist ordering, you should have a variation of this test where the "big" node is at the end of the list and check that it only boots small nodes.

    def test_node_max_nodes_two_sizes(self):
...
        self.make_daemon(want_sizes=[small, small, big, small],
                         avail_sizes=avail_sizes, max_nodes=3)

I'm not quite sure what is going on with the integration test. The test framework does have a failure mode where it crashes with "IOError: [Errno 11] Resource temporarily unavailable" I haven't gotten to the bottom of what that happens.

I think this fixes the integration tests:

diff --git a/services/nodemanager/tests/integration_test.py b/services/nodemanager/tests/integration_test.py
index 48fc3951e..508e62663 100755
--- a/services/nodemanager/tests/integration_test.py
+++ b/services/nodemanager/tests/integration_test.py
@@ -342,7 +342,6 @@ def main():
                 (r".*Cloud node (\S+) is now paired with Arvados node (\S+) with hostname (\S+)", node_paired),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Suggesting shutdown because node state is \('idle', 'open', .*\)", remaining_jobs),
-                (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown)
             ],
@@ -369,7 +368,6 @@ def main():
                 (r".*Cloud node (\S+) is now paired with Arvados node (\S+) with hostname (\S+)", node_paired),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Suggesting shutdown because node state is \('idle', 'open', .*\)", remaining_jobs),
-                (r".*ComputeNodeMonitorActor\..*\.([^[]*).*Not eligible for shut down because node state is \('busy', 'open', .*\)", node_busy),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*ComputeNodeShutdownActor\..*\.([^[]*).*Shutdown success", node_shutdown),
                 (r".*sending request", jobs_req),

Updates at b17321254
Test run: https://ci.curoverse.com/job/developer-run-tests/667/

• Updated integration tests
• Updated squeue call comment & list ordering
• Added test to check wishlist ordering

Lucas Di Pentima wrote:

Updates at b17321254
Test run: https://ci.curoverse.com/job/developer-run-tests/667/

• Updated integration tests
• Updated squeue call comment & list ordering
• Added test to check wishlist ordering

This LGTM, thanks.