Bug #13931
closed[CWL] size field not accessible on arvados
Description
Getting the cwl error.
2018-07-30T14:55:20.765294319Z cwltool ERROR: Execution failed: ../../lib/cwl/workflow.json:1:26: Expression evaluation error: 2018-07-30T14:55:20.765294319Z ../../lib/cwl/workflow.json:1:26: Syntax error in parameter reference 'inputs.fastq1.size': inputs.fastq1.size does not contain key 'size'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.
cwl script as follows
cwlVersion: v1.0 class: CommandLineTool inputs: fastq1: File outputs: out: stdout baseCommand: echo arguments: - $(inputs.fastq1.size) stdout: size.txt
Version
/data-sdd/home/jiayong/envs/acr/bin/arvados-cwl-runner 1.1.4.20180720151136, arvados-python-client 1.1.4.20180720151136, cwltool 1.0.20180615183820
Related issues
Updated by Jiayong Li about 6 years ago
- Description updated (diff)
- Status changed from In Progress to New
Updated by Jiayong Li about 6 years ago
Updated by Peter Amstutz about 6 years ago
This is a bug in a-c-r, it is failing to fill in file sizes for inputs supplied as keep references.
Updated by Peter Amstutz about 6 years ago
- Assigned To set to Peter Amstutz
- Target version set to 2018-08-15 Sprint
Updated by Peter Amstutz about 6 years ago
13931-cwl-size @ d94ed391d56f17f23e98a8e116849a44cac27e3b
https://ci.curoverse.com/view/Developer/job/developer-run-tests/838/
Corresponding work in cwltool: https://github.com/common-workflow-language/cwltool/pull/855
Updated by Tom Clegg about 6 years ago
Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.
Updated by Peter Amstutz about 6 years ago
Tom Clegg wrote:
Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.
Oops. I guess it is good that the tests are sensitive to that.
Updated by Peter Amstutz about 6 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz about 6 years ago
This exposed a different bug with URI quoting filenames:
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr Traceback (most recent call last): 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 140, in run 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr runner.arv_executor(t, job_order_object, args, logger=logger) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 559, in arv_executor 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr runtimeContext) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/arvados_cwl/arvtool.py", line 42, in job 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr builder = self._init_job(joborder, builderargs) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 594, in _init_job 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr fill_in_defaults(self.tool[u"inputs"], job, fs_access) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 392, in fill_in_defaults 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr visit_class(job, ("File",), functools.partial(add_sizes, fsaccess)) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 223, in visit_class 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr visit_class(rec[d], cls, op) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 221, in visit_class 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr op(rec) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 369, in add_sizes 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr obj["size"] = fsaccess.size(obj["location"]) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr File "/usr/lib/python2.7/dist-packages/arvados_cwl/fsaccess.py", line 149, in size 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr raise IOError(errno.EINVAL, "Not a path to a file %s" % (fn)) 2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr IOError: [Errno 22] Not a path to a file keep:6bf7c0d0f95d3ca89ce7a1d64ea4ff0a+56/item%20%231.txt
Updated by Peter Amstutz about 6 years ago
- Status changed from In Progress to Resolved
Updated by Tom Clegg about 6 years ago
- Related to Bug #14074: a-c-r does not work with input keep locators containing '#' characters added