Bug #4736
closed
Timeout value connect must be an int or float
Description
In pipeline_instances/qr1hi-d1hrv-fvgr0m5dikkozqc#Log, this keeps popping up:
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr Exception in thread Thread-21:
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr Traceback (most recent call last):
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/lib/python2.7/threading.py", line 810, in bootstrap_inner
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self.run()
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 369, in run
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self.run_with_limiter(limiter)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 381, in run_with_limiter
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout=self.args.get('timeout', None)))
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 337, in put
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout=timeout)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr return request('put', url, data=data, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr return session.request(method=method, url=url, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr resp = self.send(prep, **send_kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr r = adapter.send(request, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 313, in send
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout = TimeoutSauce(connect=timeout, read=timeout)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 88, in __init
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self._connect = self._validate_timeout(connect, 'connect')
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 119, in _validate_timeout
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr "int or float." % (name, value))
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr ValueError: Timeout value connect was (2, 300), but it must be an int or float.
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr Exception in thread Thread-24:
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr Traceback (most recent call last):
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/lib/python2.7/threading.py", line 810, in bootstrap_inner
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self.run()
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 369, in run
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self.run_with_limiter(limiter)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 381, in run_with_limiter
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout=self.args.get('timeout', None)))
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 337, in put
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout=timeout)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr return request('put', url, data=data, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr return session.request(method=method, url=url, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr resp = self.send(prep, **send_kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr r = adapter.send(request, **kwargs)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 313, in send
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr timeout = TimeoutSauce(connect=timeout, read=timeout)
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 88, in __init
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr self._connect = self._validate_timeout(connect, 'connect')
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 119, in _validate_timeout
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr "int or float." % (name, value))
Mon Dec 8 15:32:09 2014 qr1hi-8i9sb-zil9f8j0ifz9j97 27192 3 stderr ValueError: Timeout value connect was (2, 300), but it must be an int or float.
The thread number keeps changing.
Updated by Tim Pierce about 10 years ago
This error is caused when arvados-python-client is running with an old version of the requests library. requests-2.3.0 is too old to support the timeout behavior we need. The SDK installer specifies requests>=2.4.
This output appears to be generated from run-command on the compute node, outside of the job's docker image:
12/8/2014 10:28:44 AM compute29 1 task-print 1 run-command: cutadapt completed with exit code 0 (success)
12/8/2014 10:28:44 AM compute29 1 task-print 1 run-command: the following output files will be saved to keep:
12/8/2014 10:28:44 AM compute29 1 task-print 1 run-command: 1420452418 ./qr1hi-ot0gb-jw3p0jhk4t8gm8m
12/8/2014 10:28:44 AM compute29 1 task-print 1 run-command: start writing output to keep
12/8/2014 10:28:45 AM compute29 1 task-print 1 Exception in thread Thread-3:
12/8/2014 10:28:45 AM compute29 1 task-print 1 Traceback (most recent call last):
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
12/8/2014 10:28:45 AM compute29 1 task-print 1 self.run()
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 369, in run
12/8/2014 10:28:45 AM compute29 1 task-print 1 self.run_with_limiter(limiter)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 381, in run_with_limiter
12/8/2014 10:28:45 AM compute29 1 task-print 1 timeout=self.args.get('timeout', None)))
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 337, in put
12/8/2014 10:28:45 AM compute29 1 task-print 1 timeout=timeout)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 99, in put
12/8/2014 10:28:45 AM compute29 1 task-print 1 return request('put', url, data=data, **kwargs)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
12/8/2014 10:28:45 AM compute29 1 task-print 1 return session.request(method=method, url=url, **kwargs)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
12/8/2014 10:28:45 AM compute29 1 task-print 1 resp = self.send(prep, **send_kwargs)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
12/8/2014 10:28:45 AM compute29 1 task-print 1 r = adapter.send(request, **kwargs)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 313, in send
12/8/2014 10:28:45 AM compute29 1 task-print 1 timeout = TimeoutSauce(connect=timeout, read=timeout)
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 88, in __init__
12/8/2014 10:28:45 AM compute29 1 task-print 1 self._connect = self._validate_timeout(connect, 'connect')
12/8/2014 10:28:45 AM compute29 1 task-print 1 File "/usr/local/lib/python2.7/dist-packages/requests-2.3.0-py2.7.egg/requests/packages/urllib3/util/timeout.py", line 119, in _validate_timeout
12/8/2014 10:28:45 AM compute29 1 task-print 1 "int or float." % (name, value))
12/8/2014 10:28:45 AM compute29 1 task-print 1 ValueError: Timeout value connect was (2, 300), but it must be an int or float.
Can we confirm that the compute node is installing an SDK that includes commit d1cdd7ed, and that puppet is not reinstalling an older version of the Python requests library over the one we have already installed?
Updated by Brett Smith about 10 years ago
- Status changed from New to Resolved
Tim Pierce wrote:
This output appears to be generated from run-command on the compute node, outside of the job's docker image:
run-command acts as the Crunch script, and runs inside any specified Docker image. (One hint from the logs is that the output is labeled task-print; if it was coming from Crunch itself, it would say crunch or something else.)
Can we confirm that the compute node is installing an SDK that includes commit
d1cdd7ed, and that puppet is not reinstalling an older version of the Python requests library over the one we have already installed?
The compute nodes have the correct version and have since at least November 24 according to timestamps. Puppet is not doing anything to manage the requests library directly.
Bryan confirmed on IRC that updating the library is his Docker image resolved the issue, so closing this.