|
$ arvados-cwl-runner --debug msa.cwl msa_arv.yaml
|
|
INFO /home/michael/src/2020-covid-19-bh/env3.7/bin/arvados-cwl-runner 2.1.0.dev20200407192516, arvados-python-client 2.0.1, cwltool 3.0.20200324120055
|
|
INFO Resolved 'msa.cwl' to 'file:///home/michael/src/2020-covid-19-bh/msa/msa.cwl'
|
|
DEBUG Creating collection reader for 2a91d4d78e99ce298b380799f5aeae87+579
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using collection cache size 256 MiB
|
|
DEBUG ENTER jobiter 1587747830.3973134
|
|
DEBUG EXIT jobiter 1587747830.398803 0.0014896392822265625
|
|
DEBUG ENTER run 1587747830.3988857
|
|
INFO Uploading Docker image arvados/jobs:2.1.0.dev20200407192516
|
|
2020-04-24 19:03:54 arvados.arv_put[223608] INFO: Creating new cache file at /home/michael/.cache/arvados/arv-put/bd0e10119db97007dbc105c8dcc5aeea
|
|
0M / 261M 0.0% 2020-04-24 19:03:55 arvados.keep[223608] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])} (X-Request-Id: req-knwmai3k8jkwzya8p0mc)
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: 42ccf9c5663306c85f9bf29c879a35d9+67108864: ['https://keep.lugli.arvadosapi.com:443/']
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: Pool max threads is 1
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/42ccf9c5663306c85f9bf29c879a35d9
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])}
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: 956670374e4ac80b022a71ea124ae60a+5466624: ['https://keep.lugli.arvadosapi.com:443/']
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: Pool max threads is 1
|
|
2020-04-24 19:03:55 arvados.keep[223608] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/956670374e4ac80b022a71ea124ae60a
|
|
2020-04-24 19:03:56 arvados.keep[223608] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])}
|
|
2020-04-24 19:03:56 arvados.keep[223608] DEBUG: 1205e70178f09a5b4bd0a88b327ff3bf+67108864: ['https://keep.lugli.arvadosapi.com:443/']
|
|
2020-04-24 19:03:56 arvados.keep[223608] DEBUG: Pool max threads is 1
|
|
2020-04-24 19:03:56 arvados.keep[223608] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/1205e70178f09a5b4bd0a88b327ff3bf
|
|
2020-04-24 19:04:13 arvados.keep[223608] INFO: PUT 200: 5466624 bytes in 17895.527839660645 msec (0.291 MiB/sec)
|
|
2020-04-24 19:04:13 arvados.keep[223608] DEBUG: KeepWriterThread <KeepWriterThread(Thread-7, started daemon 23025616185088)> succeeded 956670374e4ac80b022a71ea124ae60a+5466624 https://keep.lugli.arvadosapi.com:443/
|
|
2020-04-24 19:04:40 arvados.keep[223608] DEBUG: Request fail: PUT https://keep.lugli.arvadosapi.com:443/42ccf9c5663306c85f9bf29c879a35d9 => <class 'arvados.errors.HttpError'>: (413, 'HTTP/2 413 \r\n')
|
|
2020-04-24 19:04:40 arvados.keep[223608] DEBUG: Request fail: PUT 42ccf9c5663306c85f9bf29c879a35d9 => 413 Could not write sufficient replicas: [0] Put http://keep0.lugli.arvadosapi.com:25107/42ccf9c5663306c85f9bf29c879a35d9: net/http: request
|
|
|
|
2020-04-24 19:04:41 arvados.keep[223608] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])}
|
|
2020-04-24 19:04:41 arvados.keep[223608] DEBUG: 42ccf9c5663306c85f9bf29c879a35d9+67108864: ['https://keep.lugli.arvadosapi.com:443/']
|
|
2020-04-24 19:04:41 arvados.keep[223608] DEBUG: Pool max threads is 1
|
|
2020-04-24 19:04:41 arvados.keep[223608] INFO: No more Keep services to try; giving up
|
|
2020-04-24 19:04:41 arvados.arv_put[223608] WARNING: Abnormal termination:
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
|
|
block.set_state(_BufferBlock.PENDING)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
|
|
raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
|
|
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start
|
|
self._upload_files()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files
|
|
output.close(flush=False)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close
|
|
self.arvadosfile.remove_writer(self, flush)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer
|
|
self.flush()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush
|
|
self.parent._my_block_manager().commit_bufferblock(bb, sync=True)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock
|
|
raise block.error
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker
|
|
loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put
|
|
data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service")
|
|
arvados.errors.KeepWriteError: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
261M / 261M 100.0% 2020-04-24 19:04:41 arvados.cwl-runner.metrics[223608] DEBUG: EXIT run 1587747881.1374114 50.73852562904358
|
|
2020-04-24 19:04:41 cwltool[223608] ERROR: Workflow execution failed
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
|
|
block.set_state(_BufferBlock.PENDING)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
|
|
raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
|
|
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvdocker.py", line 64, in arv_docker_get_image
|
|
arvados.commands.keepdocker.main(args, stdout=sys.stderr, install_sig_handlers=False, api=api_client)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/keepdocker.py", line 506, in main
|
|
install_sig_handlers=install_sig_handlers).strip()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 1304, in main
|
|
writer.start(save_collection=not(args.stream or args.raw))
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start
|
|
self._upload_files()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files
|
|
output.close(flush=False)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close
|
|
self.arvadosfile.remove_writer(self, flush)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer
|
|
self.flush()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush
|
|
self.parent._my_block_manager().commit_bufferblock(bb, sync=True)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock
|
|
raise block.error
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker
|
|
loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put
|
|
data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service")
|
|
arvados.errors.KeepWriteError: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/runner.py", line 548, in arvados_jobs_image
|
|
return arvados_cwl.arvdocker.arv_docker_get_image(arvrunner.api, {"dockerPull": img}, True, arvrunner.project_uuid)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvdocker.py", line 80, in arv_docker_get_image
|
|
cached_lookups[dockerRequirement["dockerImageId"]] = pdh
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/schema_salad/sourceline.py", line 305, in __exit__
|
|
raise self.makeError(str(exc_value)) from exc_value
|
|
cwltool.errors.WorkflowException: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/executor.py", line 703, in arv_executor
|
|
self.start_run(runnable, runtimeContext)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/executor.py", line 232, in start_run
|
|
self.workflow_eval_lock, self.stop_polling)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/task_queue.py", line 49, in add
|
|
task()
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvcontainer.py", line 508, in run
|
|
job_spec = self.arvados_job_spec(runtimeContext)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvcontainer.py", line 408, in arvados_job_spec
|
|
"container_image": arvados_jobs_image(self.arvrunner, self.jobs_image),
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/runner.py", line 550, in arvados_jobs_image
|
|
raise Exception("Docker image %s is not available\n%s" % (img, e) )
|
|
Exception: Docker image arvados/jobs:2.1.0.dev20200407192516 is not available
|
|
failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
2020-04-24 19:04:41 cwltool[223608] ERROR: Workflow error:
|
|
Workflow did not return a result.
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/cwltool/main.py", line 1105, in main
|
|
tool, initialized_job_order_object, runtimeContext, logger=_logger
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/executor.py", line 756, in arv_executor
|
|
raise WorkflowException("Workflow did not return a result.")
|
|
cwltool.errors.WorkflowException: Workflow did not return a result.
|
|
|