Bug #16367
open
forced download of arvados/jobs to client
Description
Even when not part of the workflow :-(
$ arvados-cwl-runner msa.cwl msa_arv.yaml INFO /home/michael/src/2020-covid-19-bh/env3.7/bin/arvados-cwl-runner 2.1.0.dev20200407192516, arvados-python-client 2.0.1, cwltool 3.0.20200324120055 INFO Resolved 'msa.cwl' to 'file:///home/michael/src/2020-covid-19-bh/msa/msa.cwl' INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0 INFO Using collection cache size 256 MiB INFO ['docker', 'pull', 'arvados/jobs:2.1.0.dev20200407192516'] 2.1.0.dev20200407192516: Pulling from arvados/jobs fc7181108d40: Pull complete
Files
Updated by Michael Crusoe over 4 years ago
- File mafft.cwl mafft.cwl added
- File msa.cwl msa.cwl added
- File iqtree.cwl iqtree.cwl added
- File msa_arv.yaml msa_arv.yaml added
Adding the workflow and input job object
Updated by Michael Crusoe over 4 years ago
The upload fails, I've retried multiple times:
2020-04-24 18:55:23 arvados.arv_put[153525] INFO: Resuming upload from cache file /home/michael/.cache/arvados/arv-put/bd0e10119db97007dbc105c8dcc5aeea
0M / 261M 0.0% 2020-04-24 18:56:10 arvados.arv_put[153525] WARNING: Abnormal termination:
Traceback (most recent call last):
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
block.set_state(_BufferBlock.PENDING)
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
return orig_func(self, *args, **kwargs)
File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
Updated by Michael Crusoe over 4 years ago
- File arvados-debug.log arvados-debug.log added
I did delete ~/.cache/arvados but the error persists
Here's the --debug log
Updated by Lucas Di Pentima over 4 years ago
Michael Crusoe wrote:
The upload fails, I've retried multiple times:
[...]
arv-put's error indicates that it isn't handling (or even noticing?) the 413 error when trying to PUT a block, that may need a separate bugfix ticket.
Updated by Michael Crusoe over 4 years ago
- File not-working-venv.tar.xz not-working-venv.tar.xz added
- File working-venv.tar.xz working-venv.tar.xz added
A brand new virtualenv seems to have fixed the issue. I have the original virtualenv from two weeks ago if someone wants to play detective
Updated by Michael Crusoe over 4 years ago
The issue is back for me on biohackathon.curii.com ; I cleared the cache; refreshed my credentials ; and made a new python3.7 virtualenv
413 Could not write sufficient replicas: [0] Put http://keep0.lugli.arvadosapi.com:25107/9e67b02dfa4991fb2ca5050b7ee297a6: net/http: request
Full log is attached
Updated by