Arvados

h1. Hacking Node Manager

h2. Important dependencies

h3. libcloud

"Apache Libcloud":https://libcloud.readthedocs.org/en/latest/ gives us a consistent interface to manage compute nodes across different cloud providers.

h3. Pykka

The Node Manager uses "Pykka":http://www.pykka.org/en/latest/ to easily set up lots of small workers in a multithreaded environment.  You'll probably want to read that introduction before you get started.  The Node Manager makes heavy use of Pykka's proxies.

h2. Overview - Subscriptions

Most of the actors in the Node Manager only need to communicate to others about one kind of event:

* ArvadosNodeListMonitorActor: updated information about Arvados Node objects

* ComputeNodeListMonitorActor: updated information about compute nodes running in the cloud

* JobQueueMonitorActor: updated information about the number and sizes of compute nodes that would best satisfy the job queue

* ComputeNodeSetupActor: compute node setup is finished

* ComputeNodeShutdownActor: compute node is successfully shut down

* ComputeNodeActor: compute node is eligible for shutdown

These communications happen through subscriptions.  Each actor has a @subscribe@ method that takes an arbitrary callable object, usually a proxy method.  Those callables are called with new information whenever there's a state change.

List monitor actors also have a @subscribe_to@ method that calls the callable on every update, with information about one specific object in the response (e.g., every update about an Arvados node with a specific UUID).

Thanks to this pattern, it's rare for our code to directly use the Future objects that are returned from proxy methods.  Instead, the different actors send messages to each other about interesting state changes.  The 30,000-foot overview of the program is:

* Start the list monitor actors

* Start the NodeManagerDaemonActor.  It subscribes to those monitors.

* The daemon creates different compute node actors to manage different points of the node's lifecycle, and subscribes to their updates as well.

* When the daemon creates a ComputeNodeActor, it subscribes that new actor to updates from the list monitor about the underlying cloud and Arvados data.

See @launcher.py@, and the @update_cloud_nodes@ and @update_arvados_nodes@ methods in @daemon.py@.

h2. Test Strategy

The subscription pattern simplifies testing with mocks.  Each test starts at most one actor.  We send messages to that actor with mock data, and then check the results through a mock subscriber or client objects.  As long as you can commit to particular message semantics, this makes it possible to write well-isolated, fast tests.  @testutil.py@ provides rich mocks for different kinds of objects, as well as a Mixin class to help test actors.

The tests frequently block on the result of proxy methods—i.e., they call @proxy.method().get(self.TIMEOUT)@.  This helps ensure that we know as much as possible about the actor's state before we proceed.  It also has the benefit of keeping the tests speedy, by reducing contention for Python's global interpreter lock.  Sometimes, when the tests need to ensure that an actor has handled its own internal messages generated by an event, we send another message and block on that—conventionally either a stop message, or a noop attribute access.  This ties the tests more closely to the implementation than is ideal, but it was the only solution I could find under time pressure that ran reliably on Jenkins.

h3. Why you can't check internal message handling through the actor inbox

One strategy I tried is polling the actor's message inbox, with the plan to only proceed when it is empty.  Unfortunately, this doesn't work, because messages are removed from the inbox before handling begins.  This means that if there's one message in the inbox, and handling it will generate another message, the inbox will be empty from the time processing the first message begins and the time the generated message is queued.  By itself, this is not a reliable way to ensure that an actor will not generate and handle any more internal messages.

h2. Driver wrappers

When we start a compute node, we need to seed it with information from the associated Arvados node object.  The mechanisms to pass that information will be different for each cloud provider.  To accommodate this, there are driver classes under @arvnodeman.computenode@ that handle the translation.  They also proxy public methods from the "real" libcloud driver, so except for the @create_node@ method, you can usually use libcloud's standard interfaces on our custom drivers.

h2. Configuration

@doc/ec2.example.cfg@ has lots of comments describing what parameters are available and how they behave.  Bear in mind that settings in Cloud and Size subsections are specific to the provider named in the main Cloud section.

@doc/local.example.cfg@ lets you run a development node manager, backed by libcloud's dummy driver and your development Arvados API server.  Refer to the instructions at the top of that file.