|
$ arvados-cwl-runner msa.cwl msa_arv.yaml
|
|
INFO /home/michael/src/2020-covid-19-bh/env3.7/bin/arvados-cwl-runner 2.1.0.dev20200407192516, arvados-python-client 2.0.1, cwltool 3.0.20200324120055
|
|
INFO Resolved 'msa.cwl' to 'file:///home/michael/src/2020-covid-19-bh/msa/msa.cwl'
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
|
|
INFO Using collection cache size 256 MiB
|
|
INFO ['docker', 'pull', 'arvados/jobs:2.1.0.dev20200407192516']
|
|
2.1.0.dev20200407192516: Pulling from arvados/jobs
|
|
fc7181108d40: Pull complete
|
|
e5c973b5b8ee: Pull complete
|
|
82f5b8b26077: Pull complete
|
|
5d73d76a59bb: Pull complete
|
|
72db1e144257: Pull complete
|
|
b995ebdb747b: Pull complete
|
|
fa3c0ba3b337: Pull complete
|
|
7dcb6e3b462b: Pull complete
|
|
618929f6c3c2: Pull complete
|
|
613110052abc: Pull complete
|
|
0ea5bf6bcbfb: Pull complete
|
|
Digest: sha256:f2c1bb4d16402e7e7dbd922ee69d485dd63dcb5e308fbf8b22ee20d053416363
|
|
Status: Downloaded newer image for arvados/jobs:2.1.0.dev20200407192516
|
|
docker.io/arvados/jobs:2.1.0.dev20200407192516
|
|
INFO Uploading Docker image arvados/jobs:2.1.0.dev20200407192516
|
|
2020-04-24 18:42:14 arvados.arv_put[37450] INFO: Creating new cache file at /home/michael/.cache/arvados/arv-put/bd0e10119db97007dbc105c8dcc5aeea
|
|
0M / 261M 0.0% 2020-04-24 18:43:01 arvados.arv_put[37450] WARNING: Abnormal termination:
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
|
|
block.set_state(_BufferBlock.PENDING)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
|
|
raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
|
|
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start
|
|
self._upload_files()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files
|
|
output.close(flush=False)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close
|
|
self.arvadosfile.remove_writer(self, flush)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer
|
|
self.flush()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush
|
|
self.parent._my_block_manager().commit_bufferblock(bb, sync=True)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock
|
|
raise block.error
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker
|
|
loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put
|
|
data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service")
|
|
arvados.errors.KeepWriteError: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
261M / 261M 100.0% 2020-04-24 18:43:01 cwltool[37450] ERROR: Workflow execution failed (X-Request-Id: req-dmhpayon8ql30aj22jrk)
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
|
|
block.set_state(_BufferBlock.PENDING)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
|
|
raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
|
|
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvdocker.py", line 64, in arv_docker_get_image
|
|
arvados.commands.keepdocker.main(args, stdout=sys.stderr, install_sig_handlers=False, api=api_client)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/keepdocker.py", line 506, in main
|
|
install_sig_handlers=install_sig_handlers).strip()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 1304, in main
|
|
writer.start(save_collection=not(args.stream or args.raw))
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start
|
|
self._upload_files()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files
|
|
output.close(flush=False)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close
|
|
self.arvadosfile.remove_writer(self, flush)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer
|
|
self.flush()
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush
|
|
self.parent._my_block_manager().commit_bufferblock(bb, sync=True)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock
|
|
raise block.error
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker
|
|
loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter
|
|
return orig_func(self, *args, **kwargs)
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put
|
|
data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service")
|
|
arvados.errors.KeepWriteError: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
The above exception was the direct cause of the following exception:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/runner.py", line 548, in arvados_jobs_image
|
|
return arvados_cwl.arvdocker.arv_docker_get_image(arvrunner.api, {"dockerPull": img}, True, arvrunner.project_uuid)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvdocker.py", line 80, in arv_docker_get_image
|
|
cached_lookups[dockerRequirement["dockerImageId"]] = pdh
|
|
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/schema_salad/sourceline.py", line 305, in __exit__
|
|
raise self.makeError(str(exc_value)) from exc_value
|
|
cwltool.errors.WorkflowException: failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
|
|
During handling of the above exception, another exception occurred:
|
|
|
|
Traceback (most recent call last):
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/executor.py", line 703, in arv_executor
|
|
self.start_run(runnable, runtimeContext)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/executor.py", line 232, in start_run
|
|
self.workflow_eval_lock, self.stop_polling)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/task_queue.py", line 49, in add
|
|
task()
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvcontainer.py", line 508, in run
|
|
job_spec = self.arvados_job_spec(runtimeContext)
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/arvcontainer.py", line 408, in arvados_job_spec
|
|
"container_image": arvados_jobs_image(self.arvrunner, self.jobs_image),
|
|
File "/home/michael/src/arvados/sdk/cwl/arvados_cwl/runner.py", line 550, in arvados_jobs_image
|
|
raise Exception("Docker image %s is not available\n%s" % (img, e) )
|
|
Exception: Docker image arvados/jobs:2.1.0.dev20200407192516 is not available
|
|
failed to write 42ccf9c5663306c85f9bf29c879a35d9 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413
|
|
|
|
2020-04-24 18:43:01 cwltool[37450] ERROR: Workflow error, try again with --debug for more information:
|
|
Workflow did not return a result.
|