Bug #8669
Updated by Sarah Guthrie about 8 years ago
(02:51:59 PM) sguthrie: oh dear, what is this? arvados.errors.KeepReadError: failed to read [-----]: service https://keep.qr1hi.arvadosapi.com:443/ responded with 0 (77, 'error setting certificate verify locations:\n CAfile: /etc/pki/tls/certs/ca-bundle.crt\n CApath: none')
(02:54:16 PM) brett: So, in order to verify the Keep proxy's SSL certificate, your client has to load a list of trusted certificate authorities.
(02:54:27 PM) brett: It looks for those at /etc/ssl/certs/ca-certificates.crt.
(02:54:43 PM) brett: If it doesn't find any there, it looks at the /etc/pki path in your error message.
(02:54:56 PM) brett: And if that fails it's supposed to fall back to a file from Python itself.
(02:55:21 PM) brett: Can you run ls -l on both the /etc/ssl path I gave, and the /etc/pki path in your error message, and paste the results?
(02:55:56 PM) sguthrie: -rw-r--r-- 1 root root 274340 Feb 26 20:47 /etc/ssl/certs/ca-certificates.crt
(02:56:31 PM) sguthrie: ls: cannot access /etc/pki/tls/certs/ca-bundle.crt: No such file or directory
(02:59:23 PM) brett: You ran those ls'es in the same environment as arv keep put? Same system and Docker container (if any)?
(02:59:28 PM) sguthrie: yep
(03:00:16 PM) brett: The error happens basically immediately, I'm assuming?
(03:00:49 PM) sguthrie: as soon as it starts trying to copy something from keep
(03:02:14 PM) brett: And it happens reliably?
(03:03:57 PM) sguthrie: 3/3
<pre>
File "/home/sguthrie/anaconda2/bin/arv-copy", line 4, in <module>
main()
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 136, in main
src_arv, dst_arv, args)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 290, in copy_pipeline_template
pt = copy_collections(pt, src, dst, args)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in copy_collections
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/collections.py", line 57, in __init__
self.__update(*args, **kwds)
File "/home/sguthrie/anaconda2/lib/python2.7/_abcoll.py", line 568, in update
for key, value in other:
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in <genexpr>
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in copy_collections
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/collections.py", line 57, in __init__
self.__update(*args, **kwds)
File "/home/sguthrie/anaconda2/lib/python2.7/_abcoll.py", line 568, in update
for key, value in other:
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in <genexpr>
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in copy_collections
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/collections.py", line 57, in __init__
self.__update(*args, **kwds)
File "/home/sguthrie/anaconda2/lib/python2.7/_abcoll.py", line 568, in update
for key, value in other:
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in <genexpr>
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in copy_collections
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/collections.py", line 57, in __init__
self.__update(*args, **kwds)
File "/home/sguthrie/anaconda2/lib/python2.7/_abcoll.py", line 568, in update
for key, value in other:
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in <genexpr>
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in copy_collections
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/collections.py", line 57, in __init__
self.__update(*args, **kwds)
File "/home/sguthrie/anaconda2/lib/python2.7/_abcoll.py", line 568, in update
for key, value in other:
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 342, in <genexpr>
for v in obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 337, in copy_collections
obj = arvados.util.portable_data_hash_pattern.sub(copy_collection_fn, obj)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 327, in copy_collection_fn
dst_col = copy_collection(src_id, src, dst, args)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/commands/arv_copy.py", line 577, in copy_collection
data = src_keep.get(word)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/retry.py", line 153, in num_retries_setter
return orig_func(self, *args, **kwargs)
File "/home/sguthrie/anaconda2/lib/python2.7/site-packages/arvados/keep.py", line 980, in get
"failed to read {}".format(loc_s), service_errors, label="service")
arvados.errors.KeepReadError: failed to read [-----]: service https://keep.qr1hi.arvadosapi.com:443/ responded with 0 (77, 'error setting certificate verify locations:\n CAfile: /etc/pki/tls/certs/ca-bundle.crt\n CApath: none')
</pre>