https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422015-03-10T13:23:27ZArvadosArvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=221212015-03-10T13:23:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22121/diff?detail_id=21287">diff</a>)</li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=221222015-03-10T13:29:45ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/22122/diff?detail_id=21288">diff</a>)</li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=221782015-03-11T14:47:41ZTom Cleggtom@curii.com
<ul><li><strong>Subject</strong> changed from <i>[Keep] Large downloads through workbench fail</i> to <i>[Workbench] Large downloads through workbench fail</i></li><li><strong>Category</strong> set to <i>Workbench</i></li></ul>Thoughts
<ul>
<li>Could this be a proxy issue? (Try bypassing nginx and downloading from Workbench directly, from inside the firewall?)</li>
<li>Anything in Workbench logs?</li>
<li>Anything in nginx logs?</li>
<li>Confirmed there's no problem retrieving the entire file with other tools?</li>
</ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=222912015-03-11T20:22:01ZWard Vandewegeward@curii.com
<ul><li><strong>Category</strong> deleted (<del><i>Workbench</i></del>)</li></ul><p>The bug appears to be in our code. Workbench does a fork (IO.popen) to call arv-get and streams the files. Nginx says in the logs:</p>
<pre>
2015/03/10 13:21:07 [error] 5544#0: *395704 upstream prematurely closed connection while reading upstream, client: 74.118.24.162, server: workbench.qr1hi.arvadosapi.com, request: "GET /collections/download/qr1hi-4zz18-b1uuzkf11kg3huv/3yfrrbhnsh4t1qyr8catlfa5q8uy2m7wscuvdrm4d485hqgy9u/lobstr_v3.0.2_hg19_ref/lobSTR_ref.fasta HTTP/1.1", upstream: "http://127.0.0.1:9000/collections/download/qr1hi-4zz18-b1uuzkf11kg3huv/3yfrrbhnsh4t1qyr8catlfa5q8uy2m7wscuvdrm4d485hqgy9u/lobstr_v3.0.2_hg19_ref/lobSTR_ref.fasta", host: "workbench.qr1hi.arvadosapi.com", referrer: "https://workbench.qr1hi.arvadosapi.com/collections/download/qr1hi-4zz18-b1uuzkf11kg3huv/3yfrrbhnsh4t1qyr8catlfa5q8uy2m7wscuvdrm4d485hqgy9u/"
</pre>
<p>There is nothing in the nginx error log for the process running on port 9000.</p>
<p>So, it looks like the IO.popen dies or arv-get dies, without logging anything in the webserver logs. This happens reliably at sizes just over 1 GiB.</p> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=222922015-03-11T20:22:13ZWard Vandewegeward@curii.com
<ul><li><strong>Category</strong> set to <i>Workbench</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=222972015-03-11T20:44:26ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>Bug Triage</i> to <i>2015-04-01 sprint</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=224622015-03-18T17:52:13ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=224632015-03-18T17:52:23ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=230802015-04-01T19:04:25ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-04-01 sprint</i> to <i>2015-04-29 sprint</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=230812015-04-01T19:04:32ZWard Vandewegeward@curii.com
<ul><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=231462015-04-02T17:01:39ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=236462015-04-18T00:46:50ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>In Progress</i></li><li><strong>Target version</strong> changed from <i>2015-04-29 sprint</i> to <i>Bug Triage</i></li></ul><p>I'm re-opening this bug.</p>
<p>The collection mentioned above downloads fine, now that we have proxy_buffering disabled. That's roughtly 1.8 GiB.</p>
<p>However - this collection (<a href="https://arvadosapi.com/qr1hi-4zz18-w0t3gbd4u8n5o9h">qr1hi-4zz18-w0t3gbd4u8n5o9h</a>) has a 16.4 fasta file in it, and.... it terminates download after roughly 1 GiB when downloaded through the browser. With arv keep get on a shell node, we get all 16.4 GiB without issues.</p> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=236552015-04-20T13:18:54ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>Actually, it appears that the original download listed in the ticket also does not download completely. It terminates after <strong>1.08GB</strong> and not all of the 1.8GB is downloaded.</p>
<p><a class="external" href="https://workbench.qr1hi.arvadosapi.com/collections/d341a6f1db391a780d694e240e95e475+3805/lobstr_v3.0.2_hg19_ref/lobSTR_ref.fasta?disposition=attachment&size=1885053904">https://workbench.qr1hi.arvadosapi.com/collections/d341a6f1db391a780d694e240e95e475+3805/lobstr_v3.0.2_hg19_ref/lobSTR_ref.fasta?disposition=attachment&size=1885053904</a></p> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=286072015-08-11T14:08:36ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Assigned To</strong> deleted (<del><i>Peter Amstutz</i></del>)</li><li><strong>Target version</strong> changed from <i>Bug Triage</i> to <i>Deferred</i></li></ul><p>I think we're very likely to deal with this via <a class="issue tracker-6 status-3 priority-4 priority-default closed parent" title="Idea: [Workbench] [Keep] collection browse/download server (Resolved)" href="https://dev.arvados.org/issues/5824">#5824</a>.</p> Arvados - Bug #5426: [Workbench] Large downloads through workbench failhttps://dev.arvados.org/issues/5426?journal_id=518922017-05-18T15:57:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Now using keep-web and planning to remove workbench download entirely.</p>