Arvados

In a GCE cluster I got the following

2016-01-14_11:52:51.45821 2016-01-14 11:52:51 arvnodeman.jobqueue[20780] DEBUG: JobQueueMonitorActor (at 44990544) sending poll
2016-01-14_11:52:51.46085 2016-01-14 11:52:51 arvnodeman.arvados_nodes[20780] DEBUG: ArvadosNodeListMonitorActor (at 44613072) sending poll
2016-01-14_11:52:51.46445 2016-01-14 11:52:51 arvnodeman.cloud_nodes[20780] DEBUG: CloudNodeListMonitorActor (at 40368976) sending poll
2016-01-14_11:52:51.62923 2016-01-14 11:52:51 arvnodeman.jobqueue[20780] DEBUG: Sending server wishlist: n1-standard-16, n1-standard-16, n1-standard-1, n1-standard-1, n1-st
andard-1, n1-standard-1, n1-standard-1, n1-standard-1
2016-01-14_11:52:51.62925 2016-01-14 11:52:51 arvnodeman.jobqueue[20780] DEBUG: JobQueueMonitorActor (at 44990544) got response with 8 items
2016-01-14_11:52:51.83040 2016-01-14 11:52:51 arvnodeman.arvados_nodes[20780] DEBUG: ArvadosNodeListMonitorActor (at 44613072) got response with 40 items
2016-01-14_11:52:53.46915 2016-01-14 11:52:53 arvnodeman.nodeup[20780] INFO: Creating cloud node with size n1-standard-1.
2016-01-14_11:52:55.25560 2016-01-14 11:52:55 arvnodeman.cloud_nodes[20780] DEBUG: CloudNodeListMonitorActor (at 40368976) got response with 9 items
2016-01-14_11:52:57.57508 2016-01-14 11:52:57 arvnodeman.nodeup[20780] WARNING: Client error: u"The resource 'projects/curoverse-qr2hi/zones/europe-west1-d/instances/comput
e-mbgiqs508488t4x-qr2hi' already exists" - waiting 16 seconds
2016-01-14_11:52:59.95241 2016-01-14 11:52:59 arvnodeman.nodedown[20780] WARNING: Client error: <LibcloudError in None 'Job did not complete in 20 seconds'> - waiting 1 sec
onds
2016-01-14_11:52:59.95735 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-32: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.95876 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-16: idle nodes 3, wishlist size 2
2016-01-14_11:52:59.95927 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-8: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.95974 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-4: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.96017 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-2: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.96229 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-1: idle nodes 6, wishlist size 6
2016-01-14_11:52:59.97551 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-32: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.97693 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-16: idle nodes 3, wishlist size 2
2016-01-14_11:52:59.97733 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-8: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.97766 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-4: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.97798 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-2: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.97986 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-1: idle nodes 6, wishlist size 6
2016-01-14_11:52:59.99072 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-32: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.99196 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-16: idle nodes 3, wishlist size 2
2016-01-14_11:52:59.99233 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-8: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.99277 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-4: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.99319 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-2: idle nodes 0, wishlist size 0
2016-01-14_11:52:59.99523 2016-01-14 11:52:59 arvnodeman.daemon[20780] DEBUG: n1-standard-1: idle nodes 6, wishlist size 6
2016-01-14_11:53:00.49188 2016-01-14 11:53:00 arvnodeman.nodedown[20780] INFO: Cloud node 11421971399360328232 shutdown cancelled: shutdown window closed.
201

later in the log:

2016-01-14_11:53:06.67924 2016-01-14 11:53:06 pykka[20780] DEBUG: Exception returned from ComputeNodeUpdateActor (urn:uuid:73c0280a-a09b-46c7-9bbd-8b557fa12dea) to caller:
2016-01-14_11:53:06.67926 Traceback (most recent call last):
2016-01-14_11:53:06.67927   File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 201, in _actor_loop
2016-01-14_11:53:06.67927     response = self._handle_receive(message)
2016-01-14_11:53:06.67927   File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 295, in _handle_receive
2016-01-14_11:53:06.67928     return callee(*message['args'], **message['kwargs'])
2016-01-14_11:53:06.67928   File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/dispatch/__init__.py", line 287, in throttle_wrapper
2016-01-14_11:53:06.67929     result = orig_func(self, *args, **kwargs)
2016-01-14_11:53:06.67929   File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/dispatch/__init__.py", line 300, in sync_node
2016-01-14_11:53:06.67929     return self._cloud.sync_node(cloud_node, arvados_node)
2016-01-14_11:53:06.67929   File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/driver/gce.py", line 152, in sync_node
2016-01-14_11:53:06.67930     method='POST', data=metadata_req)
2016-01-14_11:53:06.67930   File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 937, in async_request
2016-01-14_11:53:06.67931     response = request(**kwargs)
2016-01-14_11:53:06.67931   File "/usr/local/lib/python2.7/dist-packages/libcloud/compute/drivers/gce.py", line 120, in request
2016-01-14_11:53:06.67931     response = super(GCEConnection, self).request(*args, **kwargs)
2016-01-14_11:53:06.67932   File "/usr/local/lib/python2.7/dist-packages/libcloud/common/google.py", line 692, in request
2016-01-14_11:53:06.67932     *args, **kwargs)
2016-01-14_11:53:06.67932   File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 799, in request
2016-01-14_11:53:06.67932     response = responseCls(**kwargs)
2016-01-14_11:53:06.67933   File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 145, in __init__
2016-01-14_11:53:06.67933     self.object = self.parse_body()
2016-01-14_11:53:06.67933   File "/usr/local/lib/python2.7/dist-packages/libcloud/common/google.py", line 253, in parse_body
2016-01-14_11:53:06.67933     raise GoogleBaseError(message, self.status, code)
2016-01-14_11:53:06.67934 GoogleBaseError: u'Invalid fingerprint.'