Project

General

Profile

Actions

Bug #16367

open

forced download of arvados/jobs to client

Added by Michael Crusoe almost 4 years ago. Updated 28 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
-
Release:
Release relationship:
Auto

Description

Even when not part of the workflow :-(

$ arvados-cwl-runner msa.cwl msa_arv.yaml
INFO /home/michael/src/2020-covid-19-bh/env3.7/bin/arvados-cwl-runner 2.1.0.dev20200407192516, arvados-python-client 2.0.1, cwltool 3.0.20200324120055
INFO Resolved 'msa.cwl' to 'file:///home/michael/src/2020-covid-19-bh/msa/msa.cwl'
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using collection cache size 256 MiB
INFO ['docker', 'pull', 'arvados/jobs:2.1.0.dev20200407192516']
2.1.0.dev20200407192516: Pulling from arvados/jobs
fc7181108d40: Pull complete

Files

arvados.log (10.2 KB) arvados.log Michael Crusoe, 04/24/2020 04:47 PM
msa_arv.yaml (139 Bytes) msa_arv.yaml Michael Crusoe, 04/24/2020 04:57 PM
msa.cwl (734 Bytes) msa.cwl Michael Crusoe, 04/24/2020 04:57 PM
iqtree.cwl (5.31 KB) iqtree.cwl Michael Crusoe, 04/24/2020 04:57 PM
mafft.cwl (1.06 KB) mafft.cwl Michael Crusoe, 04/24/2020 04:57 PM
arvados-debug.log (15.1 KB) arvados-debug.log Michael Crusoe, 04/24/2020 05:05 PM
working-venv.tar.xz (12.5 MB) working-venv.tar.xz Michael Crusoe, 04/24/2020 05:52 PM
not-working-venv.tar.xz (11.8 MB) not-working-venv.tar.xz Michael Crusoe, 04/24/2020 05:52 PM
log (15.1 KB) log Michael Crusoe, 05/01/2020 04:19 PM
Actions #1

Updated by Michael Crusoe almost 4 years ago

Adding the workflow and input job object

Actions #2

Updated by Michael Crusoe almost 4 years ago

The upload fails, I've retried multiple times:

2020-04-24 18:55:23 arvados.arv_put[153525] INFO: Resuming upload from cache file /home/michael/.cache/arvados/arv-put/bd0e10119db97007dbc105c8dcc5aeea
0M / 261M 0.0% 2020-04-24 18:56:10 arvados.arv_put[153525] WARNING: Abnormal termination:
Traceback (most recent call last):
  File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 712, in commit_bufferblock
    block.set_state(_BufferBlock.PENDING)
  File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
    return orig_func(self, *args, **kwargs)
  File "/home/michael/src/2020-covid-19-bh/env3.7/lib/python3.7/site-packages/arvados/arvfile.py", line 360, in set_state
    raise StateChangeError("Invalid state change from %s to %s" % (self._state, nextstate), self._state, nextstate)
arvados.arvfile.StateChangeError: Invalid state change from 1 to 1
Actions #3

Updated by Michael Crusoe almost 4 years ago

I did delete ~/.cache/arvados but the error persists

Here's the --debug log

Actions #4

Updated by Lucas Di Pentima almost 4 years ago

  • Project changed from Arvados to 35
Actions #5

Updated by Lucas Di Pentima almost 4 years ago

  • Project changed from 35 to Arvados
Actions #6

Updated by Lucas Di Pentima almost 4 years ago

Michael Crusoe wrote:

The upload fails, I've retried multiple times:

[...]

arv-put's error indicates that it isn't handling (or even noticing?) the 413 error when trying to PUT a block, that may need a separate bugfix ticket.

Actions #7

Updated by Michael Crusoe almost 4 years ago

A brand new virtualenv seems to have fixed the issue. I have the original virtualenv from two weeks ago if someone wants to play detective

Actions #8

Updated by Michael Crusoe almost 4 years ago

The issue is back for me on biohackathon.curii.com ; I cleared the cache; refreshed my credentials ; and made a new python3.7 virtualenv

413 Could not write sufficient replicas: [0] Put http://keep0.lugli.arvadosapi.com:25107/9e67b02dfa4991fb2ca5050b7ee297a6: net/http: request

Full log is attached

Actions #9

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #10

Updated by Peter Amstutz 28 days ago

  • Target version set to Future
Actions

Also available in: Atom PDF