Project

General

Profile

Actions
Copy link

Idea #12409

closed

CWL 1.2 with conditionals

Added by Tom Morris about 7 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Peter Amstutz
Category:
CWL
Target version:
2020-03-25 Sprint
Start date:
01/01/2020
Due date:
Story points:
4.0
Release:
Arvados 2.1.0
Release relationship:
Auto

Description

Implement spec changes & cwltool support for workflow conditionals based on one of the designs described here:

https://github.com/common-workflow-language/common-workflow-language/issues/494

Once available in cwltool, we can update arvados-cwl-runner to take advantage of it.

Subtasks 1 (0 open — 1 closed)

Task #16184: Review 12409-cwl-v1.2ResolvedPeter Amstutz01/01/2020Actions

Related issues 1 (0 open — 1 closed)

Blocks Arvados Epics - Idea #15955: CWL 1.2Resolved02/12/202008/12/2020Actions
Actions
Copy link
 #1

Updated by Peter Amstutz about 7 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Peter Amstutz about 5 years ago

  • Target version changed from Arvados Future Sprints to 2020-01-15 Sprint
  • Subject changed from [cwl] cwl-tool conditional implementation to [cwl] CWL 1.2 with conditionals
Actions
Copy link
 #3

Updated by Peter Amstutz almost 5 years ago

  • Category set to CWL
  • Subject changed from [cwl] CWL 1.2 with conditionals to CWL 1.2 with conditionals
Actions
Copy link
 #4

Updated by Peter Amstutz almost 5 years ago

  • Start date set to 01/01/2020
  • Category deleted (CWL)
  • Due date set to 01/31/2020
  • Project changed from Arvados to Arvados Epics
Actions
Copy link
 #5

Updated by Peter Amstutz almost 5 years ago

Actions
Copy link
 #6

Updated by Peter Amstutz almost 5 years ago

Actions
Copy link
 #7

Updated by Peter Amstutz almost 5 years ago

Actions
Copy link
 #8

Updated by Peter Amstutz almost 5 years ago

  • Project changed from Arvados Epics to Arvados
Actions
Copy link
 #9

Updated by Peter Amstutz almost 5 years ago

  • Target version changed from 2020-01-15 Sprint to 2020-01-29 Sprint
Actions
Copy link
 #10

Updated by Peter Amstutz almost 5 years ago

  • Category set to CWL
Actions
Copy link
 #11

Updated by Peter Amstutz almost 5 years ago

  • Target version changed from 2020-01-29 Sprint to 2020-02-12 Sprint
Actions
Copy link
 #12

Updated by Peter Amstutz almost 5 years ago

  • Story points set to 4.0
  • Assigned To set to Peter Amstutz
Actions
Copy link
 #13

Updated by Peter Amstutz almost 5 years ago

  • Target version changed from 2020-02-12 Sprint to 2020-02-26 Sprint
Actions
Copy link
 #14

Updated by Peter Amstutz almost 5 years ago

  • Target version changed from 2020-02-26 Sprint to 2020-03-11 Sprint
Actions
Copy link
 #15

Updated by Peter Amstutz almost 5 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #17

Updated by Lucas Di Pentima almost 5 years ago

While trying to install a-c-r on a py3 virtualenv to run the arvbox tests, I got the following error:

error: ruamel.yaml 0.16.5 is installed but ruamel.yaml<=0.15.77,>=0.15.54 is required by {'arvados-python-client'}
Actions
Copy link
 #18

Updated by Peter Amstutz almost 5 years ago

Lucas Di Pentima wrote:

While trying to install a-c-r on a py3 virtualenv to run the arvbox tests, I got the following error:

[...]

Thanks, the Python SDK has a version pin on ruamel.yaml and I didn't change it. Fixed.

Actions
Copy link
 #19

Updated by Lucas Di Pentima almost 5 years ago

A couple more comments:

  • Scripts with set -x that I believe are for debugging purposes:
    • test_with_arvbox.sh - lines 6 & 99 (from past branches)
    • build-dev-docker-jobs-image.sh - line 82 from current branch
  • Have problems running cwl integration tests:
(venv-12409) lucas@buster:~/arvados/sdk/cwl/tests$ sh arvados-tests.sh 
Traceback (most recent call last):
  File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/schema.py", line 282, in load_and_validate
    strict_foreign_properties=strict_foreign_properties,
  File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 770, in resolve_ref
    strict_foreign_properties=strict_foreign_properties,
  File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 1196, in resolve_all
    strict_foreign_properties=strict_foreign_properties,
  File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 1418, in validate_links
    raise ValidationException("", None, errors)
schema_salad.exceptions.ValidationException: arvados-tests.yml:27:3:  checking item
                          `job` field is NoneType, expected string, list, or a dict.
arvados-tests.yml:38:3:  checking item
                          `job` field is NoneType, expected string, list, or a dict.
arvados-tests.yml:90:3:  checking item
                          `job` field is NoneType, expected string, list, or a dict.
[...]
Actions
Copy link
 #20

Updated by Lucas Di Pentima almost 5 years ago

Reviewing 5782d9e6f37bde05fb7dc06a312a5a909c7223e4

I was able to run the CWL integration tests, but got 9 failures, here's an extract:

(venv-12409) lucas@buster:~/arvados/sdk/cwl/tests$ sh arvados-tests.sh 
Test [1/34] Test directory in keep
Test [2/34] Test directory in keep
Test [3/34] Test default directory in keep
Test [4/34] Test default directory in keep
Test [5/34] Test hashes in filenames
Test [6/34] test shallow directory listing
Test [7/34] test no directory listing
Test [8/34] test deep directory listing
Test [9/34] Test arv:ReuseRequirement
Test [10/34] Test manipulating keep references with expression tools
Test [11/34] Test glob output on empty collection
Test [12/34] Test issue 13976
Test [13/34] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations
Test 13 failed: arvados-cwl-runner --disable-reuse --compute-checksum --api=containers --outdir=/tmp/tmp617rxs_d --quiet wf/runin-wf.cwl
RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations
Returned non-zero
WARNING Workflow checker warning:
wf/runin-wf.cwl:33:7: 'sleeptime' is not an input parameter of ordereddict([('class', 'Workflow'),
                      ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub'),
                      ('inputs', [ordereddict([('type', 'File'), ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/fileblub')])]),
                      ('outputs', [ordereddict([('type', 'string'), ('outputSource',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/out'),
                      ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/out')])]),
                      ('steps', [ordereddict([('in', [ordereddict([('source',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/fileblub'),
                      ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/fileblub')])]),
                      ('out',
                      ['file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/out']),
                      ('run', ordereddict([('class', 'CommandLineTool'), ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool'),
                      ('inputs', [ordereddict([('type', 'File'), ('inputBinding',
                      ordereddict([('position', 1)])), ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool/fileblub')])]),
                      ('outputs', [ordereddict([('type', 'string'), ('outputBinding',
                      ordereddict([('outputEval', '$("out")')])), ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool/out')])]),
                      ('baseCommand', 'cat')])), ('id',
                      'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1')])])]),
                      expected fileblub
ERROR [container runin-wf.cwl] (x2jbo-dz642-jee8sxpe2iipszv) error log:

  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:03.451938791Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-by-pdh by_id /tmp/crunch-run.x2jbo-dz642-jjh6a2sgjxayj8j.284390681/keep505234340]
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:04.144209457Z crunch-run Creating Docker container
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:04.235636360Z crunch-run Attaching container streams
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:04.415397883Z crunch-run Starting Docker container id '5c103b7eb0bb3546525fb6e44b5c10588a9b578b3de16f7eb18f7bcc9485b946'
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:04.556779046Z crunch-run Waiting for container to finish
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.718329012Z stderr INFO /usr/bin/cwltool 1.0.20190603140227
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.725322562Z stderr INFO Resolved 'workflow.cwl#main' to 'file:///var/spool/cwl/workflow.cwl#main'
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.822369379Z stderr ERROR Tool definition failed validation:
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.822369379Z stderr The CWL reference runner no longer supports pre CWL v1.0 documents. Supported versions are:
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.822369379Z stderr v1.0
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.822369379Z stderr v1.1
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.822369379Z stderr v1.1.0-dev1 (with --enable-dev flag only)
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:05.963152671Z crunch-run Container exited with code: 1
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:06.018476974Z crunch-run copying "/cwl.output.json" (0 bytes)
  2020-03-10T21:17:08.279914286Z stderr   2020-03-10T21:17:06.055107113Z crunch-run Complete
  2020-03-10T21:17:08.405690646Z stderr ERROR [container substep] while getting output object:
  2020-03-10T21:17:08.405690646Z stderr Traceback (most recent call last):
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/local/lib/python3.5/dist-packages/arvados_cwl/arvcontainer.py", line 364, in done
  2020-03-10T21:17:08.405690646Z stderr     outputs = done.done_outputs(self, container, "/tmp", self.outdir, "/keep")
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/local/lib/python3.5/dist-packages/arvados_cwl/done.py", line 55, in done_outputs
  2020-03-10T21:17:08.405690646Z stderr     return self.collect_outputs("keep:" + record["output"], record["exit_code"])
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/local/lib/python3.5/dist-packages/cwltool/command_line_tool.py", line 811, in collect_output_ports
  2020-03-10T21:17:08.405690646Z stderr     ret = json.load(f)
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/lib/python3.5/json/__init__.py", line 268, in load
  2020-03-10T21:17:08.405690646Z stderr     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
  2020-03-10T21:17:08.405690646Z stderr     return _default_decoder.decode(s)
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
  2020-03-10T21:17:08.405690646Z stderr     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  2020-03-10T21:17:08.405690646Z stderr   File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
  2020-03-10T21:17:08.405690646Z stderr     raise JSONDecodeError("Expecting value", s, err.value) from None
  2020-03-10T21:17:08.405690646Z stderr json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  2020-03-10T21:17:08.499524827Z stderr ERROR [step substep] Output is missing expected field file:///var/lib/cwl/workflow.json#main/substep/out
  2020-03-10T21:17:08.603126883Z stderr WARNING [step substep] completed permanentFail
  2020-03-10T21:17:08.713455729Z stderr INFO [workflow workflow.json#main] completed permanentFail
  2020-03-10T21:17:08.713455729Z stderr ERROR Overall process status is permanentFail
  2020-03-10T21:17:08.867141448Z stderr INFO Final output collection 237233ea9b49c1486aabfc86fbb7bc74+59 "Output of main" (x2jbo-4zz18-6a5guiu8nd0be0y)
  2020-03-10T21:17:09.086793907Z stderr WARNING Final process status is permanentFail
  2020-03-10T21:17:09.514881027Z crunch-run Container exited with code: 1
  2020-03-10T21:17:09.537807614Z crunch-run Complete
ERROR Overall process status is permanentFail
WARNING Final process status is permanentFail
[...]

I've launched a Jenkins job: arvados-cwl-conformance-tests: #384 but it's failing for other purposes.

Actions
Copy link
 #21

Updated by Peter Amstutz almost 5 years ago

  • Target version changed from 2020-03-11 Sprint to 2020-03-25 Sprint
Actions
Copy link
 #22

Updated by Peter Amstutz almost 5 years ago

12409-cwl-v1.2 @ cf447c1718535fe924005db53bed5011c714dd63

Preserve original version when packing a document for use with RunInSingleContainer.

developer-run-tests: #1772

Actions
Copy link
 #23

Updated by Lucas Di Pentima almost 5 years ago

Local cwl integration tests run was OK, but sdk/cwl tests failed both local and on jenkins.

Actions
Copy link
 #25

Updated by Lucas Di Pentima almost 5 years ago

Tests pass on my side, LGTM.

Actions
Copy link
 #26

Updated by Peter Amstutz almost 5 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link
 #27

Updated by Peter Amstutz about 4 years ago

  • Release set to 25
Actions
Copy link

Also available in: Atom PDF