Bug #16139

arvados-cwl-runner 2.0.0 secondaryFiles issue with CWL v1.0

Added by Bryan Cosca 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
02/10/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

bcosca2@lightning-dev1:~/issues/15996/cwl$ arvados-cwl-runner --submit --no-wait --project-uuid=su92l-j7d0g-4i75jw7iivgg3w7 --api=containers --debug gatk-haplotypecaller.cwl testyml/gatk-haplotypecaller.yml 
/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/__init__.py:17: CWLToolDeprecationWarning: 
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as the Python 2.7 version of cwltool won't be
maintained after that date.

  """,  category=CWLToolDeprecationWarning)
INFO /usr/bin/arvados-cwl-runner 2.0.0, arvados-python-client 2.0.0, cwltool 1.0.20190831161204
INFO Resolved 'gatk-haplotypecaller.cwl' to 'file:///data-sdc/bcosca2/issues/15996/cwl/gatk-haplotypecaller.cwl'
DEBUG Creating collection reader for 1e488d8a8de69c5e07b447c8e1ae059b+755
DEBUG Creating collection reader for a3af04432df3d71d22f2fe8be549ba96+5974
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using empty collection d41d8cd98f00b204e9800998ecf8427e+0
INFO Using collection cache size 256 MiB
ERROR Unhandled error:
  string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/main.py", line 833, in main
    logger=_logger)
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/executor.py", line 660, in arv_executor
    runnerjob = next(jobiter)
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/future/builtins/newnext.py", line 59, in newnext
    return iterator.next()
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/arvados_cwl/runner.py", line 649, in job
    self._init_job(job_order, runtimeContext)
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/process.py", line 718, in _init_job
    discover_secondaryFiles=getdefault(runtime_context.toplevel, False)))
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/builder.py", line 276, in bind_input
    bindings.extend(self.bind_input(f, datum[f["name"]], lead_pos=lead_pos, tail_pos=f["name"], discover_secondaryFiles=discover_secondaryFiles))
  File "/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/builder.py", line 317, in bind_input
    if "$(" in sf["pattern"] or "${" in sf["pattern"]:
TypeError: string indices must be integers, not str

My CWL is good though...

bcosca2@lightning-dev1:~/issues/15996/cwl$ arvados-cwl-runner --validate gatk-haplotypecaller.cwl 
/usr/share/python2.7/dist/python-arvados-cwl-runner/lib/python2.7/site-packages/cwltool/__init__.py:17: CWLToolDeprecationWarning: 
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.
Please upgrade your Python as the Python 2.7 version of cwltool won't be
maintained after that date.

  """,  category=CWLToolDeprecationWarning)
INFO /usr/bin/arvados-cwl-runner 2.0.0, arvados-python-client 2.0.0, cwltool 1.0.20190831161204
INFO Resolved 'gatk-haplotypecaller.cwl' to 'file:///data-sdc/bcosca2/issues/15996/cwl/gatk-haplotypecaller.cwl'
gatk-haplotypecaller.cwl is valid CWL.

gatk-haplotypecaller.cwl (596 Bytes) gatk-haplotypecaller.cwl Bryan Cosca, 02/07/2020 04:57 PM
gatk-haplotypecaller.yml (169 Bytes) gatk-haplotypecaller.yml Bryan Cosca, 02/07/2020 04:57 PM

Subtasks

Task #16143: Review 16139-cwl-secondaryfiles-fixResolvedPeter Amstutz

Associated revisions

Revision 6272cb13 (diff)
Added by Peter Amstutz 5 months ago

16139: Fix secondaryFile errors when running --submit --no-wait

When submitting, we want to preserve the original CWL version of the
document.

However, when it creates a RunnerContainer (a cwltool.Process) it
examines the input interface and expects it to be in the 1.1 data
model. But if we preserve/reload the original document, it is still
in the 1.0 data model. This causes

The code was deliberately overriding the CWL version in metadata to
make this work. This results in the problem reported on this ticket.

The fix is to maintain both the updated and preserved documents, and
use them appropriately where they are expected.

refs #16139

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 04332347
Added by Peter Amstutz 5 months ago

Merge branch '16139-cwl-secondaryfiles-fix'

fixes #16139

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#2 Updated by Bryan Cosca 5 months ago

I changed the cwlVersion to v1.1 and it submitted successfully.

#3 Updated by Bryan Cosca 5 months ago

  • Subject changed from arvados-cwl-runner 2.0.0 secondaryFiles issue to arvados-cwl-runner 2.0.0 secondaryFiles issue with CWL v1.0

#4 Updated by Peter Amstutz 5 months ago

  • Target version set to 2020-02-12 Sprint
  • Assigned To set to Peter Amstutz
  • Status changed from New to In Progress

#6 Updated by Lucas Di Pentima 5 months ago

This LGTM, thanks!

#7 Updated by Peter Amstutz 5 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#8 Updated by Peter Amstutz 5 months ago

  • Release set to 22

Also available in: Atom PDF