https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422017-01-24T19:34:04ZArvadosArvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=475802017-01-24T19:34:04ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>2017-02-15 sprint</i></li></ul> Arvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=479322017-02-01T20:33:01ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=479822017-02-02T14:02:51ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=480542017-02-03T19:45:38ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Branch <code>10968-arvput-resume-progress</code> at: <a class="changeset" title="10968: Added a notification when uploading at least one directory, just to let know the user that..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/ef328d9143c65c72c37194272f90fabc3ec45e0a">ef328d9</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/154/">https://ci.curoverse.com/job/developer-run-tests/154/</a></p>
There were 2 steps that take too long when uploading a big collection of files:
<ol>
<li>The <code>expected_bytes</code> calculation (when there are a lot of files on disk to be uploaded)</li>
<li>If resuming a partial upload, <code>arv-put</code> will check if there's some previously uploaded file on the cached collection that is not on disk on the current run. This was taking too much time, and was fixed on <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [arv-put] job resume too slow & uninformative (Resolved)" href="https://dev.arvados.org/issues/10932">#10932</a></li>
</ol>
<p>The updates on this branch include an optimization on the progress reporting and a message to notify the user when it's calculating the <code>expected_bytes</code> count on a potential big file list.<br />The optimization on progress reporting is about disabling unnecessary checkpointing when the upload hasn't started yet, and using the <code>skipped_bytes</code> count to show a progress indication with the said count vs total.<br />The final effect is that when resuming, the progress indicator will start from 0% to whatever percentage was left on the previous run, updating every second to provide maximum feedback until the real upload begins, changing to "checkpointing mode" every minute like before.</p> Arvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=480642017-02-03T22:00:55ZTom Cleggtom@curii.com
<ul></ul><p>LGTM, thanks! (Might as well fix errant space at EOL in comment, though... <a class="wiki-page" href="https://dev.arvados.org/projects/arvados/wiki/Coding_Standards#Source-code-formatting">Coding Standards</a>)</p> Arvados - Idea #10968: [arv-put] Notify user that resume cache is being used & provide progress indicationhttps://dev.arvados.org/issues/10968?journal_id=480672017-02-03T22:40:06ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset arvados|commit:aaa45b09de0e9437743fce53d7c0bf8165074b5e.</p>