https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422017-07-08T20:56:57ZArvadosArvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=533262017-07-08T20:56:57ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/53326/diff?detail_id=51364">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=533752017-07-11T17:26:28ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>11948-cwl-conformance-fix @ <a class="changeset" title="11948: Strip out default values which reference unavailable files. Passes CWL conformance test "..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/d1a7f9691ff2b9d77106f953c714b1455b228c64">d1a7f9691ff2b9d77106f953c714b1455b228c64</a></p>
<p>The main change in this branch is to accommodate a new CWL conformance test. This test checks that when a parameter has a default value that is a File or Directory, the default File is not required to exist provided some other File is assigned for the input parameter.</p>
<p>The implementation approach is to scan the document for "default" blocks, and check for Files under "default", and then check if those files exist. If the files do not exist, it deletes the "default" block (so that downstream processing will require a value for that parameter) and removes it from the list of files to try and upload.</p> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=533982017-07-11T21:01:24ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Some questions:</p>
<ul>
<li>File sdk/cwl/arvados_cwl/runner.py
<ul>
<li>Line 53: Could that <code>if</code> be changed to an <code>elif</code>, or there’s some case when a variable is a <code>list</code> and also a <code>dict</code>? </li>
<li>Lines 123-128: If I understand it correctly, is an in-place list modification, and I think could be written like this, what do you think?<br /><pre>
sc[:] = [x for x in sc if x[“location”] != f[“location”]]
</pre></li>
<li>Line 116: Why is <code>remove</code> a list and not simply a boolean var?</li>
</ul></li>
</ul> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=534022017-07-11T22:07:52ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Some questions:</p>
<ul>
<li>File sdk/cwl/arvados_cwl/runner.py
<ul>
<li>Line 53: Could that <code>if</code> be changed to an <code>elif</code>, or there’s some case when a variable is a <code>list</code> and also a <code>dict</code>?</li>
</ul></li>
</ul>
</blockquote>
<p>You're right. Fixed.</p>
<blockquote>
<ul>
<li>Lines 123-128: If I understand it correctly, is an in-place list modification, and I think could be written like this, what do you think?</li>
</ul>
</blockquote>
<p>That's a good one. Fixed.</p>
<blockquote>
<ul>
<li>Line 116: Why is <code>remove</code> a list and not simply a boolean var?</li>
</ul>
</blockquote>
<p>Python scopes are a little weird. When you define a function inside a function, you can access variables in the outer scope, but assignment only modifies the local scope. So setting "removed = True" wouldn't change the value of the outer "removed" variable. Instead, I use a list to add a level of indirection (sort of like a pointer).</p>
<p>Now @ <a class="changeset" title="11948: if -> elif style fix Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curovers..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/2feec8cbac7ef28a47f3c3a9d071b070ef38cb6e">2feec8cbac7ef28a47f3c3a9d071b070ef38cb6e</a></p>
<a name="Unit-testing"></a>
<h2 >Unit testing<a href="#Unit-testing" class="wiki-anchor">¶</a></h2>
<pre>
arvbox start test --only sdk/cwl
</pre>
<a name="Conformance-tests"></a>
<h2 >Conformance tests<a href="#Conformance-tests" class="wiki-anchor">¶</a></h2>
<p>To set up, create a virtualenv on your host, install the CWL SDKs from the branch, start arvbox, then build a arvados/job Docker image:</p>
<pre>
virtualenv venv && . venv/bin/activate
cd ~/arvados/sdk/cwl && python setup.py install
arvbox restart dev
export ARVADOS_API_HOST=
export ARVADOS_API_TOKEN=
cd ~/arvados/build && WORKSPACE=$HOME/arvados ./build-dev-docker-jobs-image.sh
</pre>
<p>Now you can clone the repository with the CWL tests and run them:</p>
<pre>
git clone https://github.com/common-workflow-language/common-workflow-language.git
cd common-workflow-language
./run_test.sh RUNNER=arvados-cwl-runner EXTRA="--api=containers --compute-checksum --disable-reuse"
</pre> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=534112017-07-12T17:11:14ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Local unit test worked ok on my end.<br />Updates LGTM.<br />Conformance tests couldn't be run because I wasn't able to start arvbox, just letting you know instead of blocking you because of my problem with arvbox.</p> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=535672017-07-19T18:45:07ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>2017-07-19 sprint</i> to <i>2017-08-02 sprint</i></li></ul> Arvados - Bug #11948: Fix failing CWL conformance testshttps://dev.arvados.org/issues/11948?journal_id=536132017-07-19T19:44:18ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul>