Bug #13931

[CWL] size field not accessible on arvados

Added by Jiayong Li over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/06/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Release relationship:
Auto

Description

Getting the cwl error.

2018-07-30T14:55:20.765294319Z cwltool ERROR: Execution failed: ../../lib/cwl/workflow.json:1:26: Expression evaluation error:
2018-07-30T14:55:20.765294319Z ../../lib/cwl/workflow.json:1:26: Syntax error in parameter reference 'inputs.fastq1.size': inputs.fastq1.size does not contain key 'size'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.

cwl script as follows

cwlVersion: v1.0
class: CommandLineTool
inputs:
  fastq1: File
outputs:
  out: stdout
baseCommand: echo
arguments:
  - $(inputs.fastq1.size)
stdout: size.txt

Version

/data-sdd/home/jiayong/envs/acr/bin/arvados-cwl-runner 1.1.4.20180720151136,
arvados-python-client 1.1.4.20180720151136, cwltool 1.0.20180615183820


Subtasks

Task #13949: Review 13931-cwl-sizeResolvedTom Clegg

Task #14023: Review 13931-uri-unquoteResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #14074: a-c-r does not work with input keep locators containing '#' charactersResolved

Associated revisions

Revision 61611952
Added by Peter Amstutz over 2 years ago

Merge branch '13931-cwl-size' refs #13931

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 38c4ce8b
Added by Peter Amstutz over 2 years ago

Merge branch '13931-cwltool-deps' refs #13931

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 2a298d6c (diff)
Added by Peter Amstutz over 2 years ago

Fix version of 'typing' dep refs #13931

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 4019ca8e (diff)
Added by Peter Amstutz over 2 years ago

Fix version dependency on 'typing' refs #13931

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 14c7497f
Added by Peter Amstutz over 2 years ago

Merge branch '13931-uri-unquote' refs #13931

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Jiayong Li over 2 years ago

  • Status changed from New to In Progress

#2 Updated by Jiayong Li over 2 years ago

  • Description updated (diff)
  • Status changed from In Progress to New

#4 Updated by Peter Amstutz over 2 years ago

This is a bug in a-c-r, it is failing to fill in file sizes for inputs supplied as keep references.

#5 Updated by Peter Amstutz over 2 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2018-08-15 Sprint

#7 Updated by Tom Clegg over 2 years ago

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

#8 Updated by Peter Amstutz over 2 years ago

Tom Clegg wrote:

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

Oops. I guess it is good that the tests are sensitive to that.

#12 Updated by Peter Amstutz over 2 years ago

  • Status changed from New to In Progress

#14 Updated by Peter Amstutz over 2 years ago

This exposed a different bug with URI quoting filenames:

2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr Traceback (most recent call last):
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 140, in run
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runner.arv_executor(t, job_order_object, args, logger=logger)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 559, in arv_executor
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runtimeContext)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/arvtool.py", line 42, in job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     builder = self._init_job(joborder, builderargs)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 594, in _init_job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     fill_in_defaults(self.tool[u"inputs"], job, fs_access)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 392, in fill_in_defaults
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(job, ("File",), functools.partial(add_sizes, fsaccess))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 223, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(rec[d], cls, op)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 221, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     op(rec)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 369, in add_sizes
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     obj["size"] = fsaccess.size(obj["location"])
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/fsaccess.py", line 149, in size
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     raise IOError(errno.EINVAL, "Not a path to a file %s" % (fn))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr IOError: [Errno 22] Not a path to a file keep:6bf7c0d0f95d3ca89ce7a1d64ea4ff0a+56/item%20%231.txt

#15 Updated by Tom Clegg over 2 years ago

#16 Updated by Peter Amstutz over 2 years ago

  • Status changed from In Progress to Resolved

#17 Updated by Tom Clegg over 2 years ago

  • Related to Bug #14074: a-c-r does not work with input keep locators containing '#' characters added

#18 Updated by Ward Vandewege over 2 years ago

  • Release set to 13

Also available in: Atom PDF