INFO /home/michael/src/graph-genome_pipeline/env3.7/bin/arvados-cwl-runner 2.0.2, arvados-python-client 2.0.2, cwltool 1.0.20190831161204 INFO Resolved 'pipeline.cwl' to 'file:///home/michael/src/graph-genome_pipeline/pipeline.cwl' INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Uploading Docker image odgi:latest 2020-05-01 18:17:57 arvados.arv_put[2834845] INFO: Creating new cache file at /home/michael/.cache/arvados/arv-put/2a6bf09eff2e86660b3fbca89a3abf54 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])} (X-Request-Id: req-dssm64ub883yybg7q2dc) 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: 677632652e2f78c4496689acf44ff7f3+2930176: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: 9e67b02dfa4991fb2ca5050b7ee297a6+67108864: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/677632652e2f78c4496689acf44ff7f3 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/9e67b02dfa4991fb2ca5050b7ee297a6 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])} 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: cf50cd619180f63980dc964652cd7060+67108864: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:17:58 arvados.keep[2834845] DEBUG: Request: PUT https://keep.lugli.arvadosapi.com:443/cf50cd619180f63980dc964652cd7060 2020-05-01 18:18:03 arvados.keep[2834845] INFO: PUT 200: 2930176 bytes in 5781.822443008423 msec (0.483 MiB/sec) 2020-05-01 18:18:03 arvados.keep[2834845] DEBUG: KeepWriterThread succeeded 677632652e2f78c4496689acf44ff7f3+2930176 https://keep.lugli.arvadosapi.com:443/ 2020-05-01 18:18:43 arvados.keep[2834845] DEBUG: Request fail: PUT https://keep.lugli.arvadosapi.com:443/9e67b02dfa4991fb2ca5050b7ee297a6 => : (413, 'HTTP/2 413 \r\n') 2020-05-01 18:18:43 arvados.keep[2834845] DEBUG: Request fail: PUT 9e67b02dfa4991fb2ca5050b7ee297a6 => 413 Could not write sufficient replicas: [0] Put http://keep0.lugli.arvadosapi.com:25107/9e67b02dfa4991fb2ca5050b7ee297a6: net/http: request 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: Request fail: PUT https://keep.lugli.arvadosapi.com:443/cf50cd619180f63980dc964652cd7060 => : (413, 'HTTP/2 413 \r\n') 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: Request fail: PUT cf50cd619180f63980dc964652cd7060 => 413 Could not write sufficient replicas: [0] Put http://keep0.lugli.arvadosapi.com:25107/cf50cd619180f63980dc964652cd7060: net/http: request 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])} 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: 9e67b02dfa4991fb2ca5050b7ee297a6+67108864: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:18:44 arvados.keep[2834845] INFO: No more Keep services to try; giving up 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: {'lugli-bi6l4-76544aa6dc97b808': OrderedDict([('href', '/keep_services/lugli-bi6l4-76544aa6dc97b808'), ('kind', 'arvados#keepService'), ('etag', '44f8ew7lqh8lexeyflhpsytlc'), ('uuid', 'lugli-bi6l4-76544aa6dc97b808'), ('owner_uuid', 'lugli-tpzed-000000000000000'), ('created_at', '2020-04-14T19:53:21.720990000Z'), ('modified_by_client_uuid', None), ('modified_by_user_uuid', 'lugli-tpzed-000000000000000'), ('modified_at', '2020-04-14T19:53:21.720990000Z'), ('service_host', 'keep.lugli.arvadosapi.com'), ('service_port', 443), ('service_ssl_flag', True), ('service_type', 'proxy'), ('read_only', False), ('_service_root', 'https://keep.lugli.arvadosapi.com:443/')])} 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: cf50cd619180f63980dc964652cd7060+67108864: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:18:44 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:18:44 arvados.keep[2834845] INFO: No more Keep services to try; giving up 2020-05-01 18:18:44 arvados.arv_put[2834845] WARNING: Abnormal termination: Traceback (most recent call last): File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock block.set_state(_BufferBlock.PENDING) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate) arvados.arvfile.StateChangeError: Invalid state change from 1 to 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start self._upload_files() File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files output.close(flush=False) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close self.arvadosfile.remove_writer(self, flush) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer self.flush() File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush self.parent._my_block_manager().commit_bufferblock(bb, sync=True) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock raise block.error File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service") arvados.errors.KeepWriteError: failed to write cf50cd619180f63980dc964652cd7060 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413 2020-05-01 18:18:45 arvados.keep[2834845] DEBUG: 275d4da2dd53ba32942125209c285158+67108864: ['https://keep.lugli.arvadosapi.com:443/'] 2020-05-01 18:18:45 arvados.keep[2834845] DEBUG: Pool max threads is 1 2020-05-01 18:18:45 cwltool[2834845] ERROR: Workflow error: tools/odgi/odgi_bin.cwl:15:5: failed to write cf50cd619180f63980dc964652cd7060 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413 Traceback (most recent call last): File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock block.set_state(_BufferBlock.PENDING) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate) arvados.arvfile.StateChangeError: Invalid state change from 1 to 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/arvdocker.py", line 64, in arv_docker_get_image arvados.commands.keepdocker.main(args, stdout=sys.stderr, install_sig_handlers=False, api=api_client) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/keepdocker.py", line 506, in main install_sig_handlers=install_sig_handlers).strip() File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 1304, in main writer.start(save_collection=not(args.stream or args.raw)) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 606, in start self._upload_files() File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/commands/put.py", line 835, in _upload_files output.close(flush=False) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1360, in close self.arvadosfile.remove_writer(self, flush) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1038, in remove_writer self.flush() File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 1192, in flush self.parent._my_block_manager().commit_bufferblock(bb, sync=True) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 722, in commit_bufferblock raise block.error File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 560, in _commit_bufferblock_worker loc = self._keep.put(bufferblock.buffer_view[0:bufferblock.write_pointer].tobytes(), num_retries=self.num_retries, copies=self.copies) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/retry.py", line 177, in num_retries_setter return orig_func(self, *args, **kwargs) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados/keep.py", line 1200, in put data_hash, loop.attempts_str(), copies, writer_pool.done()), service_errors, label="service") arvados.errors.KeepWriteError: failed to write cf50cd619180f63980dc964652cd7060 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/cwltool/main.py", line 833, in main logger=_logger) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/executor.py", line 579, in arv_executor merged_map = upload_workflow_deps(self, tool) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/runner.py", line 516, in upload_workflow_deps upload_docker(arvrunner, tool) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/runner.py", line 418, in upload_docker upload_docker(arvrunner, s.embedded_tool) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/runner.py", line 413, in upload_docker arvados_cwl.arvdocker.arv_docker_get_image(arvrunner.api, docker_req, True, arvrunner.project_uuid) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/arvados_cwl/arvdocker.py", line 80, in arv_docker_get_image cached_lookups[dockerRequirement["dockerImageId"]] = pdh File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/schema_salad/sourceline.py", line 298, in __exit__ raise_from(self.makeError(six.text_type(exc_value)), exc_value) File "/home/michael/src/graph-genome_pipeline/env3.7/lib/python3.7/site-packages/future/utils/__init__.py", line 403, in raise_from exec(execstr, myglobals, mylocals) File "", line 1, in cwltool.errors.WorkflowException: tools/odgi/odgi_bin.cwl:15:5: failed to write cf50cd619180f63980dc964652cd7060 after 2 attempts (wanted 2 copies but wrote 0): service https://keep.lugli.arvadosapi.com:443/ responded with 413 HTTP/2 413