https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422022-02-07T22:11:21ZArvadosArvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1006412022-02-07T22:11:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1006572022-02-08T22:32:27ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>The buggy behavior is here:</p>
<p>process.py:1109 mergedirs()</p>
<p>This function was designed to clean up the contents of "listing" or "secondaryFiles" to be suitable for materialization, where there can only be one file with a given 'basename' but there could be two or more directory literals with the same name that can be safely merged into a single one.</p>
<p>It also handles the case of two or more file entries with the same basename that don't refer to the same file.</p>
<p>However, this method is both buggy, and used inappropriately.</p>
<ul>
<li>It handles the case of two files that have different locations but the same basename, but does not check the location of Directories, so if you have two directories with different locations, one of them gets forgotten.</li>
<li>It should only be applied to "listing" and "secondaryFiles". scandeps applies it to every level regardless of whether enforcing unique basenames makes sense.</li>
</ul>
<p>scandeps should selectively apply it only to secondaryFiles and listing, or possibly not at all. I'm not sure what problem this was originally intended to solve.</p> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1006582022-02-08T22:51:22ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>I think I remember what this was for.</p>
<p>If you had two dependent files which were in subdirectories:</p>
<pre>
- class: File
basename: file1
location: foo/file1
- class: File
basename: file2
location: foo/file2
</pre>
<p>If you apply "nesting" to mimic the source directory structure, you get this:</p>
<pre>
- class: Directory
basename: foo
listing:
- class: File
basename: file1
location: foo/file1
- class: Directory
basename: foo
listing:
- class: File
basename: file2
location: foo/file2
</pre>
<p>mergedirs() turns it into this:</p>
<pre>
- class: Directory
basename: foo
listing:
- class: File
basename: file1
location: foo/file1
- class: File
basename: file2
location: foo/file2
</pre> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1006852022-02-10T14:04:29ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>The bug is in the cwltool scandeps / mergedir function:</p>
<p><a class="external" href="https://github.com/common-workflow-language/cwltool/pull/1615">https://github.com/common-workflow-language/cwltool/pull/1615</a></p> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1006982022-02-10T18:39:33ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Building a new cwltool release with the bug fix.</p>
<p>Will have a branch that updates to the new release.</p> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1007052022-02-10T21:02:14ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>18723-cwl-upload @ <a class="changeset" title="18723: Update cwltool version for scandirs fix Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pet..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0494b40a03348e584dbc255d93615a4ef01afaf9">0494b40a03348e584dbc255d93615a4ef01afaf9</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2908/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2908/">developer-run-tests: #2908 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2908" alt="" /></a></a></p> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1007392022-02-11T21:12:03ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>49</i></li></ul> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1007512022-02-14T18:49:08ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>18723-cwl-upload @ <a class="changeset" title="18723: Use setuptools console entrypoints, fix dev-jobs.dockerfile Arvados-DCO-1.1-Signed-off-by..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/e2b3ed63a292537ec7bf37a904ad2925d6afa0bf">e2b3ed63a292537ec7bf37a904ad2925d6afa0bf</a></p>
<p>Added integration test</p> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1007522022-02-14T18:51:24ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Assigned To</strong> deleted (<del><i>Peter Amstutz</i></del>)</li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>Category</strong> deleted (<del><i>CWL</i></del>)</li></ul> Arvados - Bug #18723: arvados-cwl-runner not uploading all dependencieshttps://dev.arvados.org/issues/18723?journal_id=1008162022-02-15T19:20:34ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul>