https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422018-07-30T17:51:23ZArvadosArvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=650852018-07-30T17:51:23ZJoshua Randalljr17@sanger.ac.uk
<ul><li><strong>Category</strong> set to <i>Crunch</i></li></ul><p>After implementing the initial count check with `limit: 0` (which takes ~10s on our system at present) and subsequent `count: none` on each loop iteration, on our system each batch of 100 is taking ~18s to come back (so, ~5.5 per second). Prior to the `count: none` fix, it was taking nearly 30s per batch of 100.</p>
<p>Changing the loop to use `limit: 1000` instead of the default 100 results in each batch of 1000 taking ~22s (so, ~45.5 per second).</p>
<p>Changing the loop to use `limit: 10000` results in each batch of 10000 taking ~46s to come back (so, ~217 per second).</p>
<p>I cannot test higher than this as our backlog was cleared pretty quickly when running with limit 10000 (this would have taken two hours longer to clear with the default 100 limit - although in actuality under our current conditions it would never clear the backlog as our system is submitting containers faster than it is possible for c-d-s to process them in batches of 100).</p>
<p>I would suggest making the batch size in c-d-s configurable, and/or using a larger default.</p> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=650862018-07-30T18:19:33ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>Incidentally, with batch size 10000 the processing time to clear our queued container backlog was:<br />10s get items available matching filters (fixed)<br />46s get batch of 10000 containers (~217 per second)<br />1398s lock and submit 10000 containers (~7 per second)<br />48s get batch of 10000 containers (~208 per second)<br />1032s lock and submit 10000 containers (~10 per second)<br />(followed by small batches)</p>
<p>Overall performance of c-d-s could potentially be further improved by having multiple worker goroutines handle the locking and starting of containers concurrently.</p> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=651982018-08-01T15:50:42ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>To Be Groomed</i></li></ul> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=656442018-08-13T15:23:32ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p><a class="external" href="https://github.com/curoverse/arvados/pull/77">https://github.com/curoverse/arvados/pull/77</a></p> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=657502018-08-15T16:22:01ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>To Be Groomed</i> to <i>2018-09-05 Sprint</i></li></ul> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=657562018-08-15T16:25:06ZTom Cleggtom@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li></ul> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=658582018-08-17T06:52:34ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '13933-dispatch-batch-size' closes #13933 Arvados-DCO-1.1-Signed-off-by: Tom Clegg..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/70e5c7a3c6a5860d702d5e5c219dc0f3a3696d35">arvados|70e5c7a3c6a5860d702d5e5c219dc0f3a3696d35</a>.</p> Arvados - Bug #13933: crunch-dispatch-slurm / Go SDK Dispatcher performs poorly in the presence of a large backloghttps://dev.arvados.org/issues/13933?journal_id=667842018-09-17T13:12:10ZWard Vandewegeward@curii.com
<ul><li><strong>Release</strong> set to <i>13</i></li></ul>