https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422023-05-25T15:37:21ZArvadosArvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153392023-05-25T15:37:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153402023-05-25T15:37:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>Weird delay in crunch-run finalization</i> to <i>Unexplained delay in crunch-run finalization</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153412023-05-25T15:39:02ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/115341/diff?detail_id=112122">diff</a>)</li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153422023-05-25T15:43:07ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Is it possible during this time, arv-mount/local Keepstore is uploading data that was written out by the container process before exit?</p> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153432023-05-25T16:03:47ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>On further investigation.</p>
<p>The output collection has ~4400 files, but except for the one file that was reported as being copied, it looks like these are staged to an intermediate collection and then made to appear in the output directory, and then propagated to the output collection.</p>
<p>So it seems like it is doing something that causes it to iterate over each of the 4400 files, it only needs to take 1.5s to process each file for that to add up to nearly two hours.</p>
<p>The input consists of an array of 4400 files, each file is pulled from a different collection, so I think what is happening is that it is sequentially fetching 4400 collections with manifest text.</p> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153442023-05-25T16:21:45ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>Development 2023-06-07</i> to <i>Future</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/115344/diff?detail_id=112123">diff</a>)</li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153452023-05-25T16:41:38ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Subject</strong> changed from <i>Unexplained delay in crunch-run finalization</i> to <i>Log when files from input are being propagated to output in crunch-run finalization</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153462023-05-25T17:31:39ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Story points</strong> set to <i>0.5</i></li><li><strong>Target version</strong> changed from <i>Future</i> to <i>Development 2023-06-07</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153472023-05-25T17:31:53ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153652023-05-25T18:44:55ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-1 priority-4 priority-default" href="/issues/9964">Feature #9964</a>: arvados-cwl-runner limits output data to keep using output_glob</i> added</li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153662023-05-25T18:45:26ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/115366/diff?detail_id=112153">diff</a>)</li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1153802023-05-25T20:23:52ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Category</strong> set to <i>Crunch</i></li></ul><p>20561-file-copy-logging @ <a class="changeset" title="20561: crunch-run logs files/directories propagated from keep Arvados-DCO-1.1-Signed-off-by: Pet..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/ff22334d01e09b0074be6416f9285dae8d5af565">ff22334d01e09b0074be6416f9285dae8d5af565</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/3666/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/3666/">developer-run-tests: #3666 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=3666" alt="" /></a></a></p> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1154292023-05-30T20:42:35ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>20561-file-copy-logging @ <a class="changeset" title="20561: Don't fail when inspecting a file outside the output directory Arvados-DCO-1.1-Signed-off..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/790e737b8c020f7a339ca88ee6106ae157c465fe">790e737b8c020f7a339ca88ee6106ae157c465fe</a></p>
<p><a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/3677/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/3677/">developer-run-tests: #3677 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=3677" alt="" /></a></a></p> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1155072023-06-01T14:20:27ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Peter Amstutz wrote in <a href="#note-14">#note-14</a>:</p>
<blockquote>
<p>20561-file-copy-logging @ <a class="changeset" title="20561: Don't fail when inspecting a file outside the output directory Arvados-DCO-1.1-Signed-off..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/790e737b8c020f7a339ca88ee6106ae157c465fe">790e737b8c020f7a339ca88ee6106ae157c465fe</a></p>
</blockquote>
<p>Looks good to me, thanks.</p> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1155082023-06-01T14:21:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>64</i></li></ul> Arvados - Bug #20561: Log when files from input are being propagated to output in crunch-run finalizationhttps://dev.arvados.org/issues/20561?journal_id=1155112023-06-01T14:24:18ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul>