Bug #13931: [CWL] size field not accessible on arvados - Arvados

Custom queries

All assigned issues
All issues assigned for grooming
My issues for grooming
My issues for grooming (no story pts)
Prioritized open issues

Actions

Copy link

Bug #13931

closed

[CWL] size field not accessible on arvados

Added by Jiayong Li over 5 years ago. Updated over 5 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Peter Amstutz

Category:

Target version:

2018-08-15 Sprint

Story points:

Release:

Arvados 1.2

Release relationship:

Auto

Description

Getting the cwl error.

2018-07-30T14:55:20.765294319Z cwltool ERROR: Execution failed: ../../lib/cwl/workflow.json:1:26: Expression evaluation error:
2018-07-30T14:55:20.765294319Z ../../lib/cwl/workflow.json:1:26: Syntax error in parameter reference 'inputs.fastq1.size': inputs.fastq1.size does not contain key 'size'. This could be due to using Javascript code without specifying InlineJavascriptRequirement.

cwl script as follows

cwlVersion: v1.0
class: CommandLineTool
inputs:
  fastq1: File
outputs:
  out: stdout
baseCommand: echo
arguments:
  - $(inputs.fastq1.size)
stdout: size.txt

Version

/data-sdd/home/jiayong/envs/acr/bin/arvados-cwl-runner 1.1.4.20180720151136,
arvados-python-client 1.1.4.20180720151136, cwltool 1.0.20180615183820

Subtasks 2 (0 open — 2 closed)

	Task #13949: Review 13931-cwl-size	Resolved	Tom Clegg	08/06/2018			Actions
	Task #14023: Review 13931-uri-unquote	Resolved	Peter Amstutz	08/06/2018			Actions

Related issues

Related to Arvados - Bug #14074: a-c-r does not work with input keep locators containing '#' characters

Resolved

Actions

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Jiayong Li over 5 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Jiayong Li over 5 years ago

Description updated (diff)
Status changed from In Progress to New

Actions

Copy link

Updated by Jiayong Li over 5 years ago

Related to https://projects.veritasgenetics.com/issues/8093

Actions

Copy link

Updated by Peter Amstutz over 5 years ago

This is a bug in a-c-r, it is failing to fill in file sizes for inputs supplied as keep references.

Actions

Copy link

Updated by Peter Amstutz over 5 years ago

Assigned To set to Peter Amstutz
Target version set to 2018-08-15 Sprint

Actions

Copy link

Updated by Peter Amstutz over 5 years ago

13931-cwl-size @ d94ed391d56f17f23e98a8e116849a44cac27e3b

https://ci.curoverse.com/view/Developer/job/developer-run-tests/838/

Corresponding work in cwltool: https://github.com/common-workflow-language/cwltool/pull/855

Actions

Copy link

Updated by Tom Clegg over 5 years ago

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

Actions

Copy link

Updated by Peter Amstutz over 5 years ago

Tom Clegg wrote:

Addition LGTM but it looks like various existing tests' expectations need to be updated to accommodate the new size field.

Oops. I guess it is good that the tests are sensitive to that.

Actions

Copy link

#12

Updated by Peter Amstutz over 5 years ago

Status changed from New to In Progress

Actions

Copy link

#14

Updated by Peter Amstutz over 5 years ago

This exposed a different bug with URI quoting filenames:

2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr Traceback (most recent call last):
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/crunch_script.py", line 140, in run
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runner.arv_executor(t, job_order_object, args, logger=logger)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/__init__.py", line 559, in arv_executor
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     runtimeContext)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/arvtool.py", line 42, in job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     builder = self._init_job(joborder, builderargs)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 594, in _init_job
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     fill_in_defaults(self.tool[u"inputs"], job, fs_access)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 392, in fill_in_defaults
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(job, ("File",), functools.partial(add_sizes, fsaccess))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 223, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     visit_class(rec[d], cls, op)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/utils.py", line 221, in visit_class
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     op(rec)
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/cwltool/process.py", line 369, in add_sizes
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     obj["size"] = fsaccess.size(obj["location"])
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr   File "/usr/lib/python2.7/dist-packages/arvados_cwl/fsaccess.py", line 149, in size
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr     raise IOError(errno.EINVAL, "Not a path to a file %s" % (fn))
2018-08-13_18:36:52 2tlax-8i9sb-vg3zchxctq1q4mk 29781 0 stderr IOError: [Errno 22] Not a path to a file keep:6bf7c0d0f95d3ca89ce7a1d64ea4ff0a+56/item%20%231.txt