Bug #8678

[NODEMANAGER] ComputeNodeSetupActor dies.

Added by Nico César over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
03/10/2016
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

2016-03-10_15:45:51.81829 2016-03-10 15:45:51 ComputeNodeSetupActor.2268bdfcff04[13466] INFO: Cloud node /subscriptions/a731f419-596b-4b64-a278-364e76506b06/resourceGroups/
c97qk/providers/Microsoft.Compute/virtualMachines/compute-vy79x5jddznuqnh-c97qk created.
2016-03-10_15:45:51.88281 2016-03-10 15:45:51 ComputeNodeSetupActor.2268bdfcff04[13466] INFO: c97qk-7ekkf-vy79x5jddznuqnh updated properties.
2016-03-10_15:45:51.88296 2016-03-10 15:45:51 ComputeNodeSetupActor.2268bdfcff04[13466] INFO: /subscriptions/a731f419-596b-4b64-a278-364e76506b06/resourceGroups/c97qk/provi
ders/Microsoft.Compute/virtualMachines/compute-vy79x5jddznuqnh-c97qk post-create work done.
2016-03-10_15:45:51.88470 2016-03-10 15:45:51 pykka[13466] ERROR: Unhandled exception in NodeManagerDaemonActor (urn:uuid:1c49389a-99cd-4495-9765-d1d5eec4392a):
2016-03-10_15:45:51.88471 Traceback (most recent call last):
2016-03-10_15:45:51.88472   File "/usr/lib/python2.7/dist-packages/pykka/actor.py", line 200, in _actor_loop
2016-03-10_15:45:51.88472     response = self._handle_receive(message)
2016-03-10_15:45:51.88472   File "/usr/lib/python2.7/dist-packages/pykka/actor.py", line 294, in _handle_receive
2016-03-10_15:45:51.88473     return callee(*message['args'], **message['kwargs'])
2016-03-10_15:45:51.88473   File "/usr/local/lib/python2.7/dist-packages/arvnodeman/daemon.py", line 379, in node_up
2016-03-10_15:45:51.88473     cloud_node = setup_proxy.cloud_node.get()
2016-03-10_15:45:51.88476   File "/usr/local/lib/python2.7/dist-packages/arvnodeman/baseactor.py", line 63, in __getattr__
2016-03-10_15:45:51.88477     raise AttributeError('attribute "%s" is not a callable on %s' % (name, self))
2016-03-10_15:45:51.88477 AttributeError: attribute "cloud_node" is not a callable on <ActorProxy for ComputeNodeSetupActor (urn:uuid:4b9ef2b8-a604-4e58-8137-2268bdfcff04), attr_path=()>
2016-03-10_15:45:51.88484 2016-03-10 15:45:51 ComputeNodeSetupActor.2268bdfcff04[13466] INFO: finished
2016-03-10_15:45:51.88525 2016-03-10 15:45:51 pykka[13466] DEBUG: Unregistered NodeManagerDaemonActor (urn:uuid:1c49389a-99cd-4495-9765-d1d5eec4392a)
2016-03-10_15:46:26.20571 2016-03-10 15:46:26 CloudNodeListMonitorActor.140478989115920[13466] DEBUG: sending request
2016-03-10_15:46:44.74815 2016-03-10 15:46:44 ArvadosNodeListMonitorActor.140472475259888[13466] DEBUG: sending request
2016-03-10_15:46:44.75281 2016-03-10 15:46:44 JobQueueMonitorActor.140475764400736[13466] DEBUG: sending request
current (546 KB) current Nico César, 03/10/2016 07:39 PM

Related issues

Related to Arvados - Bug #8691: [Nodemanager] NodeManagerDaemonActor dies.Resolved

Associated revisions

Revision 341066e5 (diff)
Added by Peter Amstutz over 5 years ago

Fix proxy/tell_proxy usage in _begin_node_shutdown, refs #8678

History

#1 Updated by Nico César over 5 years ago

A node gets created. But after that the death of ComputeNodeSetupActor makes that the second node isn't able to come up.

#2 Updated by Peter Amstutz over 5 years ago

I've pushed a fix, we can keep the bug open until we've confirmed it actually fixes it.

#3 Updated by Nico César over 5 years ago

Peter Amstutz wrote:

I've pushed a fix, we can keep the bug open until we've confirmed it actually fixes it.

I assume is d1de3281f023bfdbb62a172dec058caf2496224f ...
somehow Build #2513 got stuck for 17hr. re-launching it

#4 Updated by Peter Amstutz over 5 years ago

  • Target version set to 2016-03-16 sprint

#5 Updated by Peter Amstutz over 5 years ago

  • Subject changed from [NODEMANAGER] ComputeNodeSetupActor dies. to [NODEMANAGER] ComputeNodeSetupActor dies.
  • Status changed from New to In Progress

#6 Updated by Ward Vandewege over 5 years ago

  • Status changed from In Progress to Resolved

We think this was fixed in 94b8484.

Also available in: Atom PDF