Hacking Node Manager¶

Important dependencies¶

libcloud¶

Apache Libcloud gives us a consistent interface to manage compute nodes across different cloud providers.

Pykka¶

The Node Manager uses Pykka to easily set up lots of small workers in a multithreaded environment. You'll probably want to read that introduction before you get started. The Node Manager makes heavy use of Pykka's proxies.

Overview - Subscriptions¶

Most of the actors in the Node Manager only need to communicate to others about one kind of event:

• ArvadosNodeListMonitorActor: updated information about Arvados Node objects
• ComputeNodeListMonitorActor: updated information about compute nodes running in the cloud
• JobQueueMonitorActor: updated information about the number and sizes of compute nodes that would best satisfy the job queue
• ComputeNodeSetupActor: compute node setup is finished
• ComputeNodeShutdownActor: compute node is successfully shut down
• ComputeNodeActor: compute node is eligible for shutdown

These communications happen through subscriptions. Each actor has a subscribe method that takes an arbitrary callable object, usually a proxy method. Those callables are called with new information whenever there's a state change.

List monitor actors also have a subscribe_to method that calls the callable on every update, with information about one specific object in the response (e.g., every update about an Arvados node with a specific UUID).

Thanks to this pattern, it's rare for our code to directly use the Future objects that are returned from proxy methods. Instead, the different actors send messages to each other about interesting state changes. The 30,000-foot overview of the program is:

• Start the list monitor actors
• Start the NodeManagerDaemonActor. It subscribes to those monitors.
• The daemon creates different compute node actors to manage different points of the node's lifecycle, and subscribes to their updates as well.
• When the daemon creates a ComputeNodeActor, it subscribes that new actor to updates from the list monitor about the underlying cloud and Arvados data.

See launcher.py, and the update_cloud_nodes and update_arvados_nodes methods in daemon.py.

Node life cycle¶

1. JobQueueMonitorActor observes that are pending jobs in the queue, publishes "wishlist" of desired node sizes to NodeManagerDaemonActor
2. NodeManagerDaemonActor.update_server_wishlist() checks if nodes_wanted() is nonzero (meaning there is either a deficit or surplus of nodes)
3. If nodes_wanted() > 0 call NodeManagerDaemonActor.start_node()
4. NodeManagerDaemonActor.start_node() creates a ComputeNodeSetupActor and adds it to the "booting" list
5. When ComputeNodeSetupActor completes, it signals back to NodeManagerDaemonActor.node_up()
6. NodeManagerDaemonActor.node_up() removes the node from the "booting" list and puts it into the "booted" list, then starts ComputeNodeMonitorActor
7. CloudNodeListMonitorActor triggers NodeManagerDaemonActor.update_cloud_nodes(); when a node in the "booted" list shows up in cloud_nodes
   it is removed from "booted".
8. NodeManagerDaemonActor.update_cloud_nodes() pairs the cloud node with an arvados record based on:
   1. if last_ping_at is after when the cloud node was booted
   2. if the IP address of the cloud node matches the IP address in the Arvados node record
9. If the node is unpaired after the boot_fail timeout, it is shut down
10. A node is eligible for shutdown if (a) the shutdown window is open and (b) crunch_worker_state reported by API server reports 'idle'
11. To shutdown, start ComputeNodeShutdownActor
12. The node will eventually disappear from cloud_nodes reported by CloudNodeListMonitorActor

Test Strategy¶

The subscription pattern simplifies testing with mocks. Each test starts at most one actor. We send messages to that actor with mock data, and then check the results through a mock subscriber or client objects. As long as you can commit to particular message semantics, this makes it possible to write well-isolated, fast tests. testutil.py provides rich mocks for different kinds of objects, as well as a Mixin class to help test actors.

The tests frequently block on the result of proxy methods—i.e., they call proxy.method().get(self.TIMEOUT). This helps ensure that we know as much as possible about the actor's state before we proceed. It also has the benefit of keeping the tests speedy, by reducing contention for Python's global interpreter lock. Sometimes, when the tests need to ensure that an actor has handled its own internal messages generated by an event, we send another message and block on that—conventionally either a stop message, or a noop attribute access. This ties the tests more closely to the implementation than is ideal, but it was the only solution I could find under time pressure that ran reliably on Jenkins.

Why you can't check internal message handling through the actor inbox¶

One strategy I tried is polling the actor's message inbox, with the plan to only proceed when it is empty. Unfortunately, this doesn't work, because messages are removed from the inbox before handling begins. This means that if there's one message in the inbox, and handling it will generate another message, the inbox will be empty from the time processing the first message begins and the time the generated message is queued. By itself, this is not a reliable way to ensure that an actor will not generate and handle any more internal messages.

Driver wrappers¶

When we start a compute node, we need to seed it with information from the associated Arvados node object. The mechanisms to pass that information will be different for each cloud provider. To accommodate this, there are driver classes under arvnodeman.computenode that handle the translation. They also proxy public methods from the "real" libcloud driver, so except for the create_node method, you can usually use libcloud's standard interfaces on our custom drivers.

Manually testing driver wrappers¶

When the rubber hits the road, you want to be able to test the driver wrapper with the real cloud to make sure that it makes all its API calls correctly to configure compute nodes as needed. It's relatively straightforward to do that interactively, without running all of Node Manager.

First, you'll need this test harness. Save it alongside your development copy of the arvnodeman module, as drivertest.py.

# Use with `python -i` to set up a shell to interact with Node Manager
# drivers.

import sys

from pprint import pprint

from arvnodeman import config

try:
    conf_filename = sys.argv[1]
except IndexError:
    conf_filename = 'test.cfg'

myconf = config.NodeManagerConfig()
with open(conf_filename) as f:
    myconf.readfp(f)

arvnode = {
    'uuid': 'zyxwv-7ekkf-brettbrettbrett',
    'info': {'ping_secret': 'fakesecret'},
    'hostname': 'fakename',
    'domain': 'zyxwv.arvadosapi.com',
    }

driver = myconf.new_cloud_client()
rawsizes = driver.list_sizes()
sizelist = myconf.node_sizes(rawsizes)
if sizelist:
    size = sizelist[0][0]

Then:

1. Write a Node Manager configuration file for the driver you want to test. The cloud settings need to be as real as possible. Other settings need to validate but can be non-functional (e.g., you can fill in a nonexistent API server hostname and credentials)—these tests won't exercise them. DO BE CAREFUL to make sure your configuration won't interfere with production operations on a running cluster. For example, it's good to change the set of tags that identify a compute node, so a production Node Manager won't see your test nodes, and vice versa.
2. Run python -i drivertest.py YOURCONFIG.ini. It downloads basic cloud information, so it normally takes a moment for your Python prompt to appear.

Once your test interpreter is running, you have a few objects you can play with, and a few methods you'll probably want to call:

• arvnode is a fake dictionary representing an Arvados compute node API object. It has all the fields a cloud driver should need to create a cloud node.
• size is libcloud's size object for the smallest node size defined in your configuration.
• driver is the Node Manager driver wrapper for the cloud specified in your configuration.
• To create a node: cloudnode = driver.create_node(size, arvnode)
• To update the node's metadata: driver.sync_node(cloud_node, arvnode)
• To destroy the cloud node when you're done: driver.destroy_node(cloud_node)
• Of course, you should feel free to call any driver class or instance method with the data available to you.

Configuration¶

doc/ec2.example.cfg has lots of comments describing what parameters are available and how they behave. Bear in mind that settings in Cloud and Size subsections are specific to the provider named in the main Cloud section.

doc/local.example.cfg lets you run a development node manager, backed by libcloud's dummy driver and your development Arvados API server. Refer to the instructions at the top of that file.

Arvados Token¶

Node manager requires a scoped Arvados token. Create it like this as a user with an admin token:

  arv api_client_authorization create_system_auth \
    --scopes "[\"GET /arvados/v1/jobs/queue\",
               \"GET /arvados/v1/nodes\",
               \"PUT /arvados/v1/nodes\",
               \"PUT /arvados/v1/nodes/\",
               \"POST /arvados/v1/nodes\",
               \"POST /arvados/v1/nodes/\"]"