Arvados

h1. Hacking Node Manager

h2. Important dependencies

h3. libcloud

"Apache Libcloud":https://libcloud.readthedocs.org/en/latest/ gives us a consistent interface to manage compute nodes across different cloud providers.

h3. Pykka

The Node Manager uses "Pykka":http://www.pykka.org/en/latest/ to easily set up lots of small workers in a multithreaded environment.  You'll probably want to read that introduction before you get started.  The Node Manager makes heavy use of Pykka's proxies.

h2. Overview - Subscriptions

Most of the actors in the Node Manager only need to communicate to others about one kind of event:

* ArvadosNodeListMonitorActor: updated information about Arvados Node objects

* ComputeNodeListMonitorActor: updated information about compute nodes running in the cloud

* JobQueueMonitorActor: updated information about the number and sizes of compute nodes that would best satisfy the job queue

* ComputeNodeSetupActor: compute node setup is finished

* ComputeNodeShutdownActor: compute node is successfully shut down

* ComputeNodeActor: compute node is eligible for shutdown

These communications happen through subscriptions.  Each actor has a @subscribe@ method that takes an arbitrary callable object, usually a proxy method.  Those callables are called with new information whenever there's a state change.

List monitor actors also have a @subscribe_to@ method that calls the callable on every update, with information about one specific object in the response (e.g., every update about an Arvados node with a specific UUID).

Thanks to this pattern, it's rare for our code to directly use the Future objects that are returned from proxy methods.  Instead, the different actors send messages to each other about interesting state changes.  The 30,000-foot overview of the program is:

* Start the list monitor actors

* Start the NodeManagerDaemonActor.  It subscribes to those monitors.

* The daemon creates different compute node actors to manage different points of the node's lifecycle, and subscribes to their updates as well.

* When the daemon creates a ComputeNodeActor, it subscribes that new actor to updates from the list monitor about the underlying cloud and Arvados data.

See @launcher.py@, and the @update_cloud_nodes@ and @update_arvados_nodes@ methods in @daemon.py@.

h2. Node life cycle

# JobQueueMonitorActor observes that are pending jobs in the queue, publishes "wishlist" of desired node sizes to @NodeManagerDaemonActor@

# @NodeManagerDaemonActor.update_server_wishlist()@ checks if @nodes_wanted()@ is nonzero (meaning there is either a deficit or surplus of nodes)

# If @nodes_wanted() > 0@ call @NodeManagerDaemonActor.start_node()@

# @NodeManagerDaemonActor.start_node()@ creates a @ComputeNodeSetupActor@ and adds it to the "booting" list

# When @ComputeNodeSetupActor@ completes, it signals back to @NodeManagerDaemonActor.node_up()@

# @NodeManagerDaemonActor.node_up()@ removes the node from the "booting" list and puts it into the "booted" list, then starts @ComputeNodeMonitorActor@

# @CloudNodeListMonitorActor@ triggers @NodeManagerDaemonActor.update_cloud_nodes()@; when a node in the "booted" list shows up in @cloud_nodes@

 it is removed from "booted".

# @NodeManagerDaemonActor.update_cloud_nodes()@ pairs the cloud node with an arvados record based on:

## if @last_ping_at@ is after when the cloud node was booted

## if the IP address of the cloud node matches the IP address in the Arvados node record

h2. Test Strategy

The subscription pattern simplifies testing with mocks.  Each test starts at most one actor.  We send messages to that actor with mock data, and then check the results through a mock subscriber or client objects.  As long as you can commit to particular message semantics, this makes it possible to write well-isolated, fast tests.  @testutil.py@ provides rich mocks for different kinds of objects, as well as a Mixin class to help test actors.

The tests frequently block on the result of proxy methods—i.e., they call @proxy.method().get(self.TIMEOUT)@.  This helps ensure that we know as much as possible about the actor's state before we proceed.  It also has the benefit of keeping the tests speedy, by reducing contention for Python's global interpreter lock.  Sometimes, when the tests need to ensure that an actor has handled its own internal messages generated by an event, we send another message and block on that—conventionally either a stop message, or a noop attribute access.  This ties the tests more closely to the implementation than is ideal, but it was the only solution I could find under time pressure that ran reliably on Jenkins.

h3. Why you can't check internal message handling through the actor inbox

One strategy I tried is polling the actor's message inbox, with the plan to only proceed when it is empty.  Unfortunately, this doesn't work, because messages are removed from the inbox before handling begins.  This means that if there's one message in the inbox, and handling it will generate another message, the inbox will be empty from the time processing the first message begins and the time the generated message is queued.  By itself, this is not a reliable way to ensure that an actor will not generate and handle any more internal messages.

h2. Driver wrappers

When we start a compute node, we need to seed it with information from the associated Arvados node object.  The mechanisms to pass that information will be different for each cloud provider.  To accommodate this, there are driver classes under @arvnodeman.computenode@ that handle the translation.  They also proxy public methods from the "real" libcloud driver, so except for the @create_node@ method, you can usually use libcloud's standard interfaces on our custom drivers.

h2. Manually testing driver wrappers

When the rubber hits the road, you want to be able to test the driver wrapper with the real cloud to make sure that it makes all its API calls correctly to configure compute nodes as needed.  It's relatively straightforward to do that interactively, without running all of Node Manager.

First, you'll need this test harness.  Save it alongside your development copy of the @arvnodeman@ module, as @drivertest.py@.

<pre><code class="python"># Use with `python -i` to set up a shell to interact with Node Manager

# drivers.

import sys

from pprint import pprint

from arvnodeman import config

try:

    conf_filename = sys.argv[1]

except IndexError:

    conf_filename = 'test.cfg'

myconf = config.NodeManagerConfig()

with open(conf_filename) as f:

    myconf.readfp(f)

arvnode = {

    'uuid': 'zyxwv-7ekkf-brettbrettbrett',

    'info': {'ping_secret': 'fakesecret'},

    'hostname': 'fakename',

    'domain': 'zyxwv.arvadosapi.com',

    }

driver = myconf.new_cloud_client()

rawsizes = driver.list_sizes()

sizelist = myconf.node_sizes(rawsizes)

if sizelist:

    size = sizelist[0][0]

</code></pre>

Then:

# Write a Node Manager configuration file for the driver you want to test.  The cloud settings need to be as real as possible.  Other settings need to validate but can be non-functional (e.g., you can fill in a nonexistent API server hostname and credentials)—these tests won't exercise them.  DO BE CAREFUL to make sure your configuration won't interfere with production operations on a running cluster.  For example, it's good to change the set of tags that identify a compute node, so a production Node Manager won't see your test nodes, and vice versa.

# Run @python -i drivertest.py YOURCONFIG.ini@.  It downloads basic cloud information, so it normally takes a moment for your Python prompt to appear.

Once your test interpreter is running, you have a few objects you can play with, and a few methods you'll probably want to call:

* @arvnode@ is a fake dictionary representing an Arvados compute node API object.  It has all the fields a cloud driver should need to create a cloud node.

* @size@ is libcloud's size object for the smallest node size defined in your configuration.

* @driver@ is the Node Manager driver wrapper for the cloud specified in your configuration.

* To create a node: @cloudnode = driver.create_node(size, arvnode)@

* To update the node's metadata: @driver.sync_node(cloud_node, arvnode)@

* To destroy the cloud node when you're done: @driver.destroy_node(cloud_node)@

* Of course, you should feel free to call any driver class or instance method with the data available to you.

h2. Configuration

@doc/ec2.example.cfg@ has lots of comments describing what parameters are available and how they behave.  Bear in mind that settings in Cloud and Size subsections are specific to the provider named in the main Cloud section.

@doc/local.example.cfg@ lets you run a development node manager, backed by libcloud's dummy driver and your development Arvados API server.  Refer to the instructions at the top of that file.

h3. Arvados Token

Node manager requires a scoped Arvados token. Create it like this as a user with an admin token:

<pre>

  arv api_client_authorization create_system_auth \

    --scopes "[\"GET /arvados/v1/jobs/queue\",

               \"GET /arvados/v1/nodes\",

               \"PUT /arvados/v1/nodes\",

               \"PUT /arvados/v1/nodes/\",

               \"POST /arvados/v1/nodes\",

               \"POST /arvados/v1/nodes/\"]"

</pre>