https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422019-01-11T16:37:19ZArvadosArvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=705632019-01-11T16:37:19ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=705642019-01-11T16:37:26ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>New</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=705702019-01-11T19:06:02ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/70570/diff?detail_id=67673">diff</a>)</li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=705792019-01-11T21:21:47ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/70579/diff?detail_id=67684">diff</a>)</li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=705802019-01-11T21:22:40ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/70580/diff?detail_id=67685">diff</a>)</li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=715182019-02-19T15:27:31ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>To Be Groomed</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=728622019-03-27T16:04:00ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Assigned To</strong> set to <i>Eric Biagiotti</i></li><li><strong>Target version</strong> changed from <i>To Be Groomed</i> to <i>2019-04-10 Sprint</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=730962019-04-04T18:48:16ZEric Biagiotti
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=731282019-04-05T21:02:48ZEric Biagiotti
<ul></ul><p>Am able to reproduce with the following workflow:</p>
<pre><code class="yaml syntaxhl"><span class="na">class</span><span class="pi">:</span> <span class="s">ExpressionTool</span>
<span class="na">cwlVersion</span><span class="pi">:</span> <span class="s">v1.0</span>
<span class="na">inputs</span><span class="pi">:</span>
<span class="na">inputfile</span><span class="pi">:</span> <span class="s">File</span>
<span class="na">outputs</span><span class="pi">:</span>
<span class="na">out</span><span class="pi">:</span> <span class="s">Directory[]</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">InlineJavascriptRequirement</span><span class="pi">:</span> <span class="pi">{}</span>
<span class="na">expression</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">${</span>
<span class="s">var dirs = [];</span>
<span class="s">dirs.push({"class": "Directory",</span>
<span class="s">"basename": "testdir1",</span>
<span class="s">"listing": [inputs.inputfile]});</span>
<span class="s">dirs.push({"class": "Directory",</span>
<span class="s">"basename": "testdir2",</span>
<span class="s">"listing": [inputs.inputfile]});</span>
<span class="s">return {"out": dirs};</span>
<span class="s">}</span>
</code></pre>
<p>and input file:</p>
<pre><code class="yaml syntaxhl"><span class="na">inputfile</span><span class="pi">:</span>
<span class="na">class</span><span class="pi">:</span> <span class="s">File</span>
<span class="na">path</span><span class="pi">:</span> <span class="s">test.txt</span>
</code></pre> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=731902019-04-10T13:21:11ZEric Biagiotti
<ul><li><strong>Target version</strong> changed from <i>2019-04-10 Sprint</i> to <i>2019-04-24 Sprint</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=732492019-04-10T21:08:28ZEric Biagiotti
<ul></ul><p>The <code>14723-cwl-multiple-file-targets</code> branch at <a class="changeset" title="14723: Adds tests for multiple file targets and literal name resolution Arvados-DCO-1.1-Signed-o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/810fc3e7aab72aa5ae951141955b56f6d4e47cfd">810fc3e7aab72aa5ae951141955b56f6d4e47cfd</a> now maps files from their target destination instead. The exception is for literals, which store file contents instead of a resolved location.</p>
<p>I also added a test for this and found a bug where an ExpressionTool with a file array output of literals with the same name would not use name conflict resolution. I fixed this and also added a test.</p>
<p>Unit Tests: <a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests/1185/">https://ci.curoverse.com/view/Developer/job/developer-run-tests/1185/</a><br />C/I Tests: <a class="external" href="https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/89/">https://ci.curoverse.com/view/CWL/job/arvados-cwl-conformance-tests/89/</a></p> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=732682019-04-11T16:04:17ZEric Biagiotti
<ul></ul><p>Re-run of service/api tests: <a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests-services-api/1210/">https://ci.curoverse.com/view/Developer/job/developer-run-tests-services-api/1210/</a></p> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=733012019-04-11T20:39:46ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Please add a comment about how StagingPathMapper, instead of internally mapping source → target, maps targets → source.</p>
<p>Also how mapper() is overridden to maintain the behavior of mapping source → target</p>
<p>Also how literals are a special case, they are still mapped by source (identifier) → target</p>
<pre>
if v.resolved == src or (v.type == "CreateFile" and k == src):
return v
</pre>
<p>Let's make this logic exclusive:</p>
<pre>
if (v.type != "CreateFile" and v.resolved == src) or (v.type == "CreateFile" and k == src):
</pre>
<p>The code blocks at arvcontainer.py:150 and executor.py:444 are very similar if not identical in function, I wonder if they could be refactored to use the same logic.</p>
<p>Unrelated to this immediate bug, but I noticed this in executor.py:464</p>
<pre>
except IOError as e:
logger.warning("While preparing output collection: %s", e)
</pre>
<p>Seems like that should be logger.error() and/or re-raise the exception.</p> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=734752019-04-19T19:53:09ZEric Biagiotti
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>Please add a comment about how StagingPathMapper, instead of internally mapping source → target, maps targets → source.</p>
<p>Also how mapper() is overridden to maintain the behavior of mapping source → target</p>
<p>Also how literals are a special case, they are still mapped by source (identifier) → target</p>
</blockquote>
<p>Done</p>
<blockquote>
<p>[...]</p>
<p>Let's make this logic exclusive:</p>
<p>[...]</p>
</blockquote>
<p>Fixed</p>
<blockquote>
<p>The code blocks at arvcontainer.py:150 and executor.py:444 are very similar if not identical in function, I wonder if they could be refactored to use the same logic.</p>
</blockquote>
<p>I don't see a huge benefit to refactoring this at the moment, at least as a part of this bug. Maybe we can add a ticket for this?</p>
<blockquote>
<p>Unrelated to this immediate bug, but I noticed this in executor.py:464</p>
<p>[...]</p>
<p>Seems like that should be logger.error() and/or re-raise the exception.</p>
</blockquote>
<p>Fixed</p>
<p>Latest at <a class="changeset" title="Merge branch 'master' into 14723-cwl-multiple-file-targets refs #14723 Arvados-DCO-1.1-Signed-o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/454ee2b8f0385c542b6f1165a3baf2820425e1a3">454ee2b8f0385c542b6f1165a3baf2820425e1a3</a>. <br />Unit Tests: <a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests/1195/">https://ci.curoverse.com/view/Developer/job/developer-run-tests/1195/</a><br />C/I Tests: Run and passed on my machine.</p> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=734792019-04-19T20:24:33ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Eric Biagiotti wrote:</p>
<blockquote>
<p>Latest at <a class="changeset" title="Merge branch 'master' into 14723-cwl-multiple-file-targets refs #14723 Arvados-DCO-1.1-Signed-o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/454ee2b8f0385c542b6f1165a3baf2820425e1a3">454ee2b8f0385c542b6f1165a3baf2820425e1a3</a>. <br />Unit Tests: <a class="external" href="https://ci.curoverse.com/view/Developer/job/developer-run-tests/1195/">https://ci.curoverse.com/view/Developer/job/developer-run-tests/1195/</a><br />C/I Tests: Run and passed on my machine.</p>
</blockquote>
<p>LGTM</p> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=734812019-04-19T20:35:10ZEric Biagiotti
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul> Arvados - Bug #14723: [CWL] File that should appear multiple times in output collection only appears oncehttps://dev.arvados.org/issues/14723?journal_id=746652019-05-21T22:27:38ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Release</strong> set to <i>15</i></li></ul>