Arvados

https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-bn1dzf48g4jtzgq#Log

2015-05-06_13:23:07 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr [Crunch] Arvados SDK added to RUBYLIB
2015-05-06_13:23:07 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr [Crunch] Arvados SDK added to PERLLIB
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr Traceback (most recent call last):
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr File "/tmp/crunch-job/src/crunch_scripts/variantannotator", line 15, in <module>
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr input_vcf_collection = coll(input_vcf_collection_id)
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr File "/tmp/crunch-job-work/.arvados.venv/local/lib/python2.7/site-packages/arvados/collection.py", line 1064, in init
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr self._populate()
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr File "/tmp/crunch-job-work/.arvados.venv/local/lib/python2.7/site-packages/arvados/collection.py", line 1168, in _populate
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr error_via_keep))
2015-05-06_13:23:08 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr arvados.errors.NotFoundError: Failed to retrieve collection '07ad1365016e0b951867c07a89011577+15373' from either API server (<HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/07ad1365016e0b951867c07a89011577%2B15373?alt=json returned "Path not found">) or Keep (07ad1365016e0b951867c07a89011577+15373 not found: http://[keep3.qr1hi.arvadosapi.com]:25107/ responded with 403 Forbidden; http://[keep0.qr1hi.arvadosapi.com]:25107/ responded with 403 Forbidden; http://[keep2.qr1hi.arvadosapi.com]:25107/ responded with 403 Forbidden; http://[keep1.qr1hi.arvadosapi.com]:25107/ responded with 403 Forbidden).
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 stderr srun: error: compute56: task 0: Exited with exit code 1
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 child 8220 on compute56.1 exit 1 success=
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 failure (#1, permanent) after 12 seconds
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 0 task output (0 bytes):
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 Every node has failed -- giving up on this round
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 wait for last 0 children to finish
2015-05-06_13:23:09 qr1hi-8i9sb-ledk1prybpi3t71 7998 status: 0 done, 0 running, 1 todo
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 release job allocation
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 Freeze not implemented
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 collate
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 collated output manifest text to send to API server is 0 bytes with access tokens
2015-05-06_13:23:10 salloc: Job allocation 1109 has been revoked.
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 job output d41d8cd98f00b204e9800998ecf8427e+0
2015-05-06_13:23:10 qr1hi-8i9sb-ledk1prybpi3t71 7998 finish
2015-05-06_13:23:19 Traceback (most recent call last):
2015-05-06_13:23:20 File "/usr/local/bin/arv-put", line 4, in <module>
2015-05-06_13:23:20 main()
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 474, in main
2015-05-06_13:23:20 writer.finish_current_stream()
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 318, in finish_current_stream
2015-05-06_13:23:20 self.flush_data()
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 305, in flush_data
2015-05-06_13:23:20 super(ArvPutCollectionWriter, self).flush_data()
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 264, in flush_data
2015-05-06_13:23:20 copies=self.replication))
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 154, in num_retries_setter
2015-05-06_13:23:20 return orig_func(self, *args, **kwargs)
2015-05-06_13:23:20 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 968, in put
2015-05-06_13:23:20 data_hash, copies, thread_limiter.done()), service_errors, label="service")
2015-05-06_13:23:20 arvados.errors.KeepWriteError: failed to write 1241dee8c79a70beab4f73676cb89eff (wanted 2 copies but wrote 1): service http://keep2.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 100 Continue
2015-05-06_13:23:20 HTTP/1.1 503 Service Unavailable; service http://keep0.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 503 Service Unavailable
2015-05-06_13:23:20 ; service http://keep1.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 100 Continue
2015-05-06_13:23:20 HTTP/1.1 503 Service Unavailable
2015-05-06_13:23:20 qr1hi-8i9sb-7kcne1kltzp7jqz 6846 log_writer_finish: arv-put exited 1
2015-05-06_13:23:20 qr1hi-8i9sb-7kcne1kltzp7jqz 6846 log manifest is
2015-05-06_13:23:39 Traceback (most recent call last):
2015-05-06_13:23:39 File "/usr/local/bin/arv-put", line 4, in <module>
2015-05-06_13:23:39 main()
2015-05-06_13:23:39 File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 474, in main
2015-05-06_13:23:39 writer.finish_current_stream()
2015-05-06_13:23:39 File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 318, in finish_current_stream
2015-05-06_13:23:40 self.flush_data()
2015-05-06_13:23:40 File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 305, in flush_data
2015-05-06_13:23:40 super(ArvPutCollectionWriter, self).flush_data()
2015-05-06_13:23:40 File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 264, in flush_data
2015-05-06_13:23:40 copies=self.replication))
2015-05-06_13:23:40 File "/usr/local/lib/python2.7/dist-packages/arvados/retry.py", line 154, in num_retries_setter
2015-05-06_13:23:40 return orig_func(self, *args, **kwargs)
2015-05-06_13:23:40 File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 968, in put
2015-05-06_13:23:40 data_hash, copies, thread_limiter.done()), service_errors, label="service")
2015-05-06_13:23:40 arvados.errors.KeepWriteError: failed to write 7aef41c2cb250c5e5e90ec01e3741d00 (wanted 2 copies but wrote 1): service http://keep1.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 100 Continue
2015-05-06_13:23:40 HTTP/1.1 503 Service Unavailable; service http://keep2.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 100 Continue
2015-05-06_13:23:40 HTTP/1.1 503 Service Unavailable; service http://keep0.qr1hi.arvadosapi.com:25107/ responded with 503 HTTP/1.1 503 Service Unavailable
2015-05-06_13:23:40
2015-05-06_13:23:40 qr1hi-8i9sb-ledk1prybpi3t71 7998 log_writer_finish: arv-put exited 1
2015-05-06_13:23:40 qr1hi-8i9sb-ledk1prybpi3t71 7998 log manifest is