Arvados

• max queue time of any queued job/container (this should be implemented in apiserver, not nodemanager)
• number of alive compute nodes
• number of allocated compute nodes
• configured max compute nodes
• total hourly cost
• configured max hourly cost
• are we currently waiting for a node to turn off before trying again because we hit a quota?
• max idle time of any compute node
• max uptime of any compute node
• number of errors received from cloud provider by this process
• this process uptime
• number of occurrences of unpaired→shutdown transition (node was created but never pinged within configured boot-wait)

from note-5 I'll be ordering by priority for Ops Needs and the reason in parenthesis:

1. configured max compute nodes (this will help on all graphs, basically exposing the configuration )
2. idle time of all compute node (we can get the max of this, but also we need to know which idle nodes are misbehaving)
3. number of alive compute nodes and number of allocated compute nodes (this 2 metrics will give us an idea current state of costs versus actual use)
4. number of occurrences of unpaired→shutdown transition (node was created but never pinged within configured boot-wait)
5. number of errors received from cloud provider by this process
6. number of exceptions generated by all actors in this process (and successfully catched)
7. this process uptime (this is kind of irrelevant since the process will suicide if a major problem happens but also in a normal deploy. If it's easy to do, then is better to have it, otherwise just ignore)

1. total hourly cost
2. configured max hourly cost

1. max uptime of any compute node
2. are we currently waiting for a node to turn off before trying again because we hit a quota?

1. max queue time of any queued job/container (this should be implemented in apiserver, not nodemanager)

Updates at a02012dd9 - branch 12085-anm-metrics
Test run: https://ci.curoverse.com/job/developer-run-tests/671/

• max_nodes: Expose nodemanager's configuration
• actor_exceptions: Actor non fatal error counter
• cloud_errors: CLOUD_ERRORS exception counter
• boot_failures: Number of times any node goes from unpaired to shutdown
• idle_nodes: Hash with counters for every node that is on idle state, stating how many seconds is in that state. When a node leaves the idle state, it's removed from this hash (asked Nico about this behavior).

Also added tests for all the new stats.

Regarding the number of alive versus allocated nodes, the status tracker already show how many nodes are on every state, so I think that's enough.

In addition to max_nodes it should also expose the current value of node_quota.

Could we make cloud_errors more specific? Like "create_node_errors", "destroy_node_errors", "list_node_errors" ?

It looks like it is missing a call to idle_out() when a node disappears from the cloud node list?

Would it make sense to add a counter like "time_spent_idle" which is the sum of node idle times since node manager started? It might be useful to get a sense of how much time is actually wasted.

Peter Amstutz wrote:

Would it make sense to add a counter like "time_spent_idle" which is the sum of node idle times since node manager started? It might be useful to get a sense of how much time is actually wasted.

I think all agregations can be done in the tool that uses this data. Even if is a trivial thing in the code, I feel there will be turtles down the road when node manager restarts (which suicide is the mother of all fallbacks in a-n-m)

Updates at c18fb83a3
Test run: https://ci.curoverse.com/job/developer-run-tests/675/

• Added node_quota metrics.
• Splitted cloud_errors into list_nodes_errors, create_node_errors and destroy_node_errors.
• Added missing status.tracker.idle_out() call when a idle node is detected to be missing from the cloud node list.
• Added/updated related tests.

Rebased against latest master at 5fd2ed9e93670007226a1772040a966fb9dd4d22
Test run: https://ci.curoverse.com/job/developer-run-tests/676/

                if record.actor:
                    try:
                        # If it's paired and idle, stop its idle time counter
                        # before removing the monitor actor.
                        if record.actor.get_state().get() == 'idle':
                            status.tracker.idle_out(
                                record.actor.arvados_node.get()['hostname'])
                        record.actor.stop()
                    except pykka.ActorDeadError:
                        pass

Take out if record.actor.get_state().get() == 'idle': and call it unconditionally.

I believe you can use record.arvados_node["hostname"] directly intead of calling the actor.

Actually I think the whole block should go outside of "if record.actor"

if record.arvados_node:
    status.tracker.idle_out(record.arvados_node.get('hostname'))
if record.actor:
    ...

Suggestion addressed at 842c85cde
Test run: https://ci.curoverse.com/job/developer-run-tests/677/

Lucas Di Pentima wrote:

Suggestion addressed at 842c85cde
Test run: https://ci.curoverse.com/job/developer-run-tests/677/

LGTM