https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422017-10-05T19:08:19ZArvadosArvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=557792017-10-05T19:08:19ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/55779/diff?detail_id=53515">diff</a>)</li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=797742019-12-18T16:19:40ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2020-01-15 Sprint</i></li><li><strong>Subject</strong> changed from <i>[cwl] cwl-tool conditional implementation</i> to <i>[cwl] CWL 1.2 with conditionals</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801112019-12-31T21:59:16ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Category</strong> set to <i>CWL</i></li><li><strong>Subject</strong> changed from <i>[cwl] CWL 1.2 with conditionals</i> to <i>CWL 1.2 with conditionals</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801202019-12-31T22:27:35ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Start date</strong> set to <i>01/01/2020</i></li><li><strong>Category</strong> deleted (<del><i>CWL</i></del>)</li><li><strong>Due date</strong> set to <i>01/31/2020</i></li><li><strong>Project</strong> changed from <i>Arvados</i> to <i>Arvados Epics</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801222019-12-31T22:29:33ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/15955">Idea #15955</a>: CWL 1.2</i> added</li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801242019-12-31T22:29:45ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> deleted (<i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/15955">Idea #15955</a>: CWL 1.2</i>)</li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801262019-12-31T22:29:54ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/15955">Idea #15955</a>: CWL 1.2</i> added</li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=801282019-12-31T22:30:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Project</strong> changed from <i>Arvados Epics</i> to <i>Arvados</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=802052020-01-02T16:13:13ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-01-15 Sprint</i> to <i>2020-01-29 Sprint</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=802312020-01-02T19:03:44ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Category</strong> set to <i>CWL</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=806082020-01-15T15:45:32ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-01-29 Sprint</i> to <i>2020-02-12 Sprint</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=813062020-01-23T20:35:31ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Story points</strong> set to <i>4.0</i></li><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=814062020-01-29T14:49:11ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-02-12 Sprint</i> to <i>2020-02-26 Sprint</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=826252020-02-26T15:16:38ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-02-26 Sprint</i> to <i>2020-03-11 Sprint</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=826262020-02-26T15:16:48ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829222020-03-09T23:30:53ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>12409-cwl-v1.2 @ <a class="changeset" title="12409: Fix tests Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f3b02f85ce244e22033e40e59caf87228b11faee">f3b02f85ce244e22033e40e59caf87228b11faee</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1768/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1768/">developer-run-tests: #1768 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=1768" alt="" /></a></a></p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829512020-03-10T19:45:05ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>While trying to install a-c-r on a py3 virtualenv to run the arvbox tests, I got the following error:</p>
<pre>
error: ruamel.yaml 0.16.5 is installed but ruamel.yaml<=0.15.77,>=0.15.54 is required by {'arvados-python-client'}
</pre> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829522020-03-10T20:00:29ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>While trying to install a-c-r on a py3 virtualenv to run the arvbox tests, I got the following error:</p>
<p>[...]</p>
</blockquote>
<p>Thanks, the Python SDK has a version pin on ruamel.yaml and I didn't change it. Fixed.</p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829562020-03-10T20:37:00ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>A couple more comments:</p>
<ul>
<li>Scripts with <code>set -x</code> that I believe are for debugging purposes:
<ul>
<li><code>test_with_arvbox.sh</code> - lines 6 & 99 (from past branches)</li>
<li><code>build-dev-docker-jobs-image.sh</code> - line 82 from current branch</li>
</ul>
</li>
<li>Have problems running cwl integration tests:</li>
</ul>
<pre>
(venv-12409) lucas@buster:~/arvados/sdk/cwl/tests$ sh arvados-tests.sh
Traceback (most recent call last):
File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/schema.py", line 282, in load_and_validate
strict_foreign_properties=strict_foreign_properties,
File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 770, in resolve_ref
strict_foreign_properties=strict_foreign_properties,
File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 1196, in resolve_all
strict_foreign_properties=strict_foreign_properties,
File "/home/lucas/venv-12409/lib/python3.7/site-packages/schema_salad-5.0.20200220195218-py3.7.egg/schema_salad/ref_resolver.py", line 1418, in validate_links
raise ValidationException("", None, errors)
schema_salad.exceptions.ValidationException: arvados-tests.yml:27:3: checking item
`job` field is NoneType, expected string, list, or a dict.
arvados-tests.yml:38:3: checking item
`job` field is NoneType, expected string, list, or a dict.
arvados-tests.yml:90:3: checking item
`job` field is NoneType, expected string, list, or a dict.
[...]
</pre> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829702020-03-10T21:30:49ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Reviewing <a class="changeset" title="12409: Update schema-salad pin to latest Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.ams..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5782d9e6f37bde05fb7dc06a312a5a909c7223e4">5782d9e6f37bde05fb7dc06a312a5a909c7223e4</a></p>
<p>I was able to run the CWL integration tests, but got 9 failures, here's an extract:</p>
<pre>
(venv-12409) lucas@buster:~/arvados/sdk/cwl/tests$ sh arvados-tests.sh
Test [1/34] Test directory in keep
Test [2/34] Test directory in keep
Test [3/34] Test default directory in keep
Test [4/34] Test default directory in keep
Test [5/34] Test hashes in filenames
Test [6/34] test shallow directory listing
Test [7/34] test no directory listing
Test [8/34] test deep directory listing
Test [9/34] Test arv:ReuseRequirement
Test [10/34] Test manipulating keep references with expression tools
Test [11/34] Test glob output on empty collection
Test [12/34] Test issue 13976
Test [13/34] RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations
Test 13 failed: arvados-cwl-runner --disable-reuse --compute-checksum --api=containers --outdir=/tmp/tmp617rxs_d --quiet wf/runin-wf.cwl
RunInSingleContainer cwl.input.json needs to be consistent with pathmapper manipulations
Returned non-zero
WARNING Workflow checker warning:
wf/runin-wf.cwl:33:7: 'sleeptime' is not an input parameter of ordereddict([('class', 'Workflow'),
('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub'),
('inputs', [ordereddict([('type', 'File'), ('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/fileblub')])]),
('outputs', [ordereddict([('type', 'string'), ('outputSource',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/out'),
('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/out')])]),
('steps', [ordereddict([('in', [ordereddict([('source',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/fileblub'),
('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/fileblub')])]),
('out',
['file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/out']),
('run', ordereddict([('class', 'CommandLineTool'), ('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool'),
('inputs', [ordereddict([('type', 'File'), ('inputBinding',
ordereddict([('position', 1)])), ('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool/fileblub')])]),
('outputs', [ordereddict([('type', 'string'), ('outputBinding',
ordereddict([('outputEval', '$("out")')])), ('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1/subtool/out')])]),
('baseCommand', 'cat')])), ('id',
'file:///media/psf/arvados/sdk/cwl/tests/wf/runin-wf.cwl#substep/mysub/sleep1')])])]),
expected fileblub
ERROR [container runin-wf.cwl] (<a href="https://arvadosapi.com/x2jbo-dz642-jee8sxpe2iipszv">x2jbo-dz642-jee8sxpe2iipszv</a>) error log:
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:03.451938791Z crunch-run Running [arv-mount --foreground --allow-other --read-write --crunchstat-interval=10 --file-cache 268435456 --mount-by-pdh by_id /tmp/crunch-run.<a href="https://arvadosapi.com/x2jbo-dz642-jjh6a2sgjxayj8j">x2jbo-dz642-jjh6a2sgjxayj8j</a>.284390681/keep505234340]
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:04.144209457Z crunch-run Creating Docker container
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:04.235636360Z crunch-run Attaching container streams
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:04.415397883Z crunch-run Starting Docker container id '5c103b7eb0bb3546525fb6e44b5c10588a9b578b3de16f7eb18f7bcc9485b946'
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:04.556779046Z crunch-run Waiting for container to finish
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.718329012Z stderr INFO /usr/bin/cwltool 1.0.20190603140227
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.725322562Z stderr INFO Resolved 'workflow.cwl#main' to 'file:///var/spool/cwl/workflow.cwl#main'
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.822369379Z stderr ERROR Tool definition failed validation:
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.822369379Z stderr The CWL reference runner no longer supports pre CWL v1.0 documents. Supported versions are:
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.822369379Z stderr v1.0
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.822369379Z stderr v1.1
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.822369379Z stderr v1.1.0-dev1 (with --enable-dev flag only)
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:05.963152671Z crunch-run Container exited with code: 1
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:06.018476974Z crunch-run copying "/cwl.output.json" (0 bytes)
2020-03-10T21:17:08.279914286Z stderr 2020-03-10T21:17:06.055107113Z crunch-run Complete
2020-03-10T21:17:08.405690646Z stderr ERROR [container substep] while getting output object:
2020-03-10T21:17:08.405690646Z stderr Traceback (most recent call last):
2020-03-10T21:17:08.405690646Z stderr File "/usr/local/lib/python3.5/dist-packages/arvados_cwl/arvcontainer.py", line 364, in done
2020-03-10T21:17:08.405690646Z stderr outputs = done.done_outputs(self, container, "/tmp", self.outdir, "/keep")
2020-03-10T21:17:08.405690646Z stderr File "/usr/local/lib/python3.5/dist-packages/arvados_cwl/done.py", line 55, in done_outputs
2020-03-10T21:17:08.405690646Z stderr return self.collect_outputs("keep:" + record["output"], record["exit_code"])
2020-03-10T21:17:08.405690646Z stderr File "/usr/local/lib/python3.5/dist-packages/cwltool/command_line_tool.py", line 811, in collect_output_ports
2020-03-10T21:17:08.405690646Z stderr ret = json.load(f)
2020-03-10T21:17:08.405690646Z stderr File "/usr/lib/python3.5/json/__init__.py", line 268, in load
2020-03-10T21:17:08.405690646Z stderr parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
2020-03-10T21:17:08.405690646Z stderr File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
2020-03-10T21:17:08.405690646Z stderr return _default_decoder.decode(s)
2020-03-10T21:17:08.405690646Z stderr File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
2020-03-10T21:17:08.405690646Z stderr obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2020-03-10T21:17:08.405690646Z stderr File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
2020-03-10T21:17:08.405690646Z stderr raise JSONDecodeError("Expecting value", s, err.value) from None
2020-03-10T21:17:08.405690646Z stderr json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-03-10T21:17:08.499524827Z stderr ERROR [step substep] Output is missing expected field file:///var/lib/cwl/workflow.json#main/substep/out
2020-03-10T21:17:08.603126883Z stderr WARNING [step substep] completed permanentFail
2020-03-10T21:17:08.713455729Z stderr INFO [workflow workflow.json#main] completed permanentFail
2020-03-10T21:17:08.713455729Z stderr ERROR Overall process status is permanentFail
2020-03-10T21:17:08.867141448Z stderr INFO Final output collection 237233ea9b49c1486aabfc86fbb7bc74+59 "Output of main" (<a href="https://arvadosapi.com/x2jbo-4zz18-6a5guiu8nd0be0y">x2jbo-4zz18-6a5guiu8nd0be0y</a>)
2020-03-10T21:17:09.086793907Z stderr WARNING Final process status is permanentFail
2020-03-10T21:17:09.514881027Z crunch-run Container exited with code: 1
2020-03-10T21:17:09.537807614Z crunch-run Complete
ERROR Overall process status is permanentFail
WARNING Final process status is permanentFail
[...]
</pre>
<p>I've launched a Jenkins job: <a class="external" href="https://ci.arvados.org/job/arvados-cwl-conformance-tests/384/"<a href="https://ci.arvados.org/job/arvados-cwl-conformance-tests/384/">arvados-cwl-conformance-tests: #384 <img src="https://ci.arvados.org/buildStatus/icon?job=arvados-cwl-conformance-tests&build=384" alt="" /></a></a> but it's failing for other purposes.</p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=829892020-03-11T13:39:35ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2020-03-11 Sprint</i> to <i>2020-03-25 Sprint</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=830692020-03-12T19:33:09ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>12409-cwl-v1.2 @ <a class="changeset" title="12409: Fix packing bug in cwltool, bump dependency on cwltool Arvados-DCO-1.1-Signed-off-by: Pet..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cf447c1718535fe924005db53bed5011c714dd63">cf447c1718535fe924005db53bed5011c714dd63</a></p>
<p>Preserve original version when packing a document for use with RunInSingleContainer.</p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1772/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1772/">developer-run-tests: #1772 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=1772" alt="" /></a></a></p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=830772020-03-12T22:02:28ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Local cwl integration tests run was OK, but <code>sdk/cwl</code> tests failed both local and on jenkins.</p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=831152020-03-17T21:40:08ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>12409-cwl-v1.2 @ <a class="changeset" title="12409: Bump cwltool version for more bugfixing. The tests pass, and the tests confirm that the p..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/60ed2df387ca0ee304bf8d2e40f2199184375b29">60ed2df387ca0ee304bf8d2e40f2199184375b29</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1778/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/1778/">developer-run-tests: #1778 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=1778" alt="" /></a></a></p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=831492020-03-18T19:00:40ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Tests pass on my side, LGTM.</p> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=831532020-03-18T19:17:15ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul> Arvados - Idea #12409: CWL 1.2 with conditionalshttps://dev.arvados.org/issues/12409?journal_id=876342020-10-07T02:11:36ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>25</i></li></ul>