Project

General

Profile

Actions

Bug #13931

closed

[CWL] size field not accessible on arvados

Added by Jiayong Li about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/06/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release:
Release relationship:
Auto

Description

Getting the cwl error.

2018-07-30T14:55:20.765294319Z cwltool ERROR: Execution failed: ../../lib/cwl/workflow.json:1:26: Expression evaluation error:
2018-07-30T14:55:20.765294319Z ../../lib/cwl/workflow.json:1:26: Syntax error in parameter reference 'inputs.fastq1.size': inputs.fastq1.size does not contain key 'size'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.

cwl script as follows

cwlVersion: v1.0
class: CommandLineTool
inputs:
  fastq1: File
outputs:
  out: stdout
baseCommand: echo
arguments:
  - $(inputs.fastq1.size)
stdout: size.txt

Version

/data-sdd/home/jiayong/envs/acr/bin/arvados-cwl-runner 1.1.4.20180720151136,
arvados-python-client 1.1.4.20180720151136, cwltool 1.0.20180615183820


Subtasks 2 (0 open2 closed)

Task #13949: Review 13931-cwl-sizeResolvedTom Clegg08/06/2018

Actions
Task #14023: Review 13931-uri-unquoteResolvedPeter Amstutz08/06/2018

Actions

Related issues

Related to Arvados - Bug #14074: a-c-r does not work with input keep locators containing '#' charactersResolved

Actions
Actions #1

Updated by Jiayong Li about 4 years ago

  • Status changed from New to In Progress
Actions #2

Updated by Jiayong Li about 4 years ago

  • Description updated (diff)
  • Status changed from In Progress to New
Actions #4

Updated by Peter Amstutz about 4 years ago

This is a bug in a-c-r, it is failing to fill in file sizes for inputs supplied as keep references.

Actions #5

Updated by Peter Amstutz about 4 years ago

  • Assigned To set to Peter Amstutz
  • Target version set to 2018-08-15 Sprint
Actions #7

Updated by Tom Clegg about 4 years ago

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

Actions #8

Updated by Peter Amstutz about 4 years ago

Tom Clegg wrote:

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

Oops. I guess it is good that the tests are sensitive to that.

Actions #12

Updated by Peter Amstutz about 4 years ago

  • Status changed from New to In Progress
Actions #14

Updated by Peter Amstutz about 4 years ago

This exposed a different bug with URI quoting filenames:

2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr Traceback (most recent call last):
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 140, in run
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runner.arv_executor(t, job_order_object, args, logger=logger)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 559, in arv_executor
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runtimeContext)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/arvtool.py", line 42, in job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     builder = self._init_job(joborder, builderargs)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 594, in _init_job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     fill_in_defaults(self.tool[u"inputs"], job, fs_access)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 392, in fill_in_defaults
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(job, ("File",), functools.partial(add_sizes, fsaccess))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 223, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(rec[d], cls, op)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 221, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     op(rec)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 369, in add_sizes
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     obj["size"] = fsaccess.size(obj["location"])
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/fsaccess.py", line 149, in size
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     raise IOError(errno.EINVAL, "Not a path to a file %s" % (fn))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr IOError: [Errno 22] Not a path to a file keep:6bf7c0d0f95d3ca89ce7a1d64ea4ff0a+56/item%20%231.txt
Actions #15

Updated by Tom Clegg about 4 years ago

Actions #16

Updated by Peter Amstutz about 4 years ago

  • Status changed from In Progress to Resolved
Actions #17

Updated by Tom Clegg about 4 years ago

  • Related to Bug #14074: a-c-r does not work with input keep locators containing '#' characters added
Actions #18

Updated by Ward Vandewege about 4 years ago

  • Release set to 13
Actions

Also available in: Atom PDF