Actions
Bug #6666
closed
[Node Manager] Sometimes raises GoogleBaseError: u'Invalid fingerprint.'
Status:
Duplicate
Priority:
Normal
Assigned To:
-
Category:
Node Manager
Target version:
-
Story points:
-
Description
arvados-node-manager 0.1.20150511215048 gave exception
2015-07-17_14:38:50.20116 2015-07-17 14:38:50 pykka[30669] DEBUG: Exception returned from ComputeNodeUpdateActor (urn:uuid:5d00d4ba-330e-4b08-88ac-5794ac0d371d) to caller: 2015-07-17_14:38:50.20120 Traceback (most recent call last): 2015-07-17_14:38:50.20120 File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 200, in _actor_loop 2015-07-17_14:38:50.20121 response = self._handle_receive(message) 2015-07-17_14:38:50.20121 File "/usr/local/lib/python2.7/dist-packages/pykka/actor.py", line 294, in _handle_receive 2015-07-17_14:38:50.20122 return callee(*message['args'], **message['kwargs']) 2015-07-17_14:38:50.20122 File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/dispatch/__init__.py", line 252, in throttle_wrapper 2015-07-17_14:38:50.20123 result = orig_func(self, *args, **kwargs) 2015-07-17_14:38:50.20123 File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/dispatch/__init__.py", line 265, in sync_node 2015-07-17_14:38:50.20124 return self._cloud.sync_node(cloud_node, arvados_node) 2015-07-17_14:38:50.20124 File "/usr/local/lib/python2.7/dist-packages/arvnodeman/computenode/driver/gce.py", line 144, in sync_node 2015-07-17_14:38:50.20125 method='POST', data=metadata_req) 2015-07-17_14:38:50.20125 File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 874, in async_request 2015-07-17_14:38:50.20126 response = request(**kwargs) 2015-07-17_14:38:50.20127 File "/usr/local/lib/python2.7/dist-packages/libcloud/common/google.py", line 684, in request 2015-07-17_14:38:50.20127 *args, **kwargs) 2015-07-17_14:38:50.20127 File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 736, in request 2015-07-17_14:38:50.20128 response = responseCls(**kwargs) 2015-07-17_14:38:50.20128 File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 119, in __init__ 2015-07-17_14:38:50.20128 self.object = self.parse_body() 2015-07-17_14:38:50.20128 File "/usr/local/lib/python2.7/dist-packages/libcloud/common/google.py", line 249, in parse_body 2015-07-17_14:38:50.20129 raise GoogleBaseError(message, self.status, code) 2015-07-17_14:38:50.20129 GoogleBaseError: u'Invalid fingerprint.' 2015-07-17_14:38:56.68637 2015-07-17 14:38:56 arvnodeman.arvados_nodes[30669] DEBUG: ArvadosNodeListMonitorActor (at 22100688) sending poll 2015-07-17_14:38:56.68725 2015-07-17 14:38:56 arvnodeman.jobqueue[30669] DEBUG: JobQueueMonitorActor (at 29545168) sending poll 2015-07-17_14:38:56.73487 2015-07-17 14:38:56 arvnodeman.jobqueue[30669] DEBUG: Sending server wishlist: n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8, n1-standard-8 2015-07-17_14:38:56.73495 2015-07-17 14:38:56 arvnodeman.jobqueue[30669] DEBUG: JobQueueMonitorActor (at 29545168) got response with 17 items 2015-07-17_14:38:56.97197 2015-07-17 14:38:56 arvnodeman.arvados_nodes[30669] DEBUG: ArvadosNodeListMonitorActor (at 22100688) got response with 143 items 2015-07-17_14:39:31.90599 2015-07-17 14:39:31 arvnodeman.cloud_nodes[30669] WARNING: CloudNodeListMonitorActor (at 22127120) got error: The read operation timed out - waiting 120 seconds
Updated by Brett Smith almost 9 years ago
- Subject changed from Exception GoogleBaseError: u'Invalid fingerprint.' to [Node Manager] Sometimes raises GoogleBaseError: u'Invalid fingerprint.'
- Category set to Node Manager
The exception is ugly, but FWIW, this exception appearing in the logs is not a huge deal. It means the node's metadata has already changed since the last version Node Manager saw when it submitted its own change request. Node Manager will retry its request when this happens, until it succeeds in making the change it wants. See the comment added in fff9d822 for very nitty-gritty details.
Actions