https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422016-09-08T20:34:40ZArvadosArvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=428642016-09-08T20:34:40ZTom Cleggtom@curii.com
<ul></ul><p>This looks like the index response is truncated, e.g., due to a network problem or keepstore crash. We should fix the error reporting so it's more obvious if/when this happens.</p> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=544922017-08-29T13:29:02ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=625852018-05-10T14:26:24ZTom Cleggtom@curii.com
<ul></ul><p>We have seen similar errors caused by timeouts; keep-balance uses a client that times out 5 minutes after starting a request, even if data has been arriving the whole time. In the case of indexing, it would be more appropriate to time out only if the connection is silent for 5 minutes. The next best thing would be to have a longer/configurable timeout.</p>
<p>In any case we should also display the connection error instead of (or in addition to) the "truncated input" error, so the operator can tell what's happening.</p> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=625932018-05-10T15:36:17ZTom Cleggtom@curii.com
<ul></ul><p>9918-index-error @ <a class="changeset" title="9918: Don't mask http/network error with response body format error. Arvados-DCO-1.1-Signed-off-..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/7da2c48e181298f5b5d1691de76603b9e92ec9d6">7da2c48e181298f5b5d1691de76603b9e92ec9d6</a></p> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=625942018-05-10T15:36:40ZTom Cleggtom@curii.com
<ul><li><strong>Category</strong> set to <i>Keep</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2018-05-23 Sprint</i></li></ul> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=628122018-05-18T19:52:22ZTom Cleggtom@curii.com
<ul></ul><p>9918-index-timeout @ <a class="changeset" title="9918: Add RequestTimeout config for keep-balance. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tcle..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/012677d2d3fb4571da4a48ea49eae156f28bf6af">012677d2d3fb4571da4a48ea49eae156f28bf6af</a> adds a RequestTimeout config.</p>
<p>This isn't as good as a "timeout if connection goes silent for N seconds" but at least it's better than a hard-coded 5 minute timeout.</p>
<pre><code class="yaml syntaxhl"><span class="na">Client</span><span class="pi">:</span>
<span class="na">APIHost</span><span class="pi">:</span> <span class="s">zzzzz.arvadosapi.com:443</span>
<span class="na">AuthToken</span><span class="pi">:</span> <span class="s">xyzzy</span>
<span class="na">Insecure</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">KeepServiceTypes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">disk</span>
<span class="na">RunPeriod</span><span class="pi">:</span> <span class="s">600s</span>
<span class="na">CollectionBatchSize</span><span class="pi">:</span> <span class="m">100000</span>
<span class="na">CollectionBuffers</span><span class="pi">:</span> <span class="m">1000</span>
<span class="na">RequestTimeout</span><span class="pi">:</span> <span class="s">30m</span>
</code></pre> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=628522018-05-22T14:51:44ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=628542018-05-22T15:11:39ZWard Vandewegeward@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>9918-index-timeout @ <a class="changeset" title="9918: Add RequestTimeout config for keep-balance. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tcle..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/012677d2d3fb4571da4a48ea49eae156f28bf6af">012677d2d3fb4571da4a48ea49eae156f28bf6af</a> adds a RequestTimeout config.</p>
<p>This isn't as good as a "timeout if connection goes silent for N seconds" but at least it's better than a hard-coded 5 minute timeout.</p>
<p>[...]</p>
</blockquote>
<p>9918-index-timeout @ 012677d2d3fb4571da4a48ea49eae156f28bf6af LGTM, I've tested it and the settings work.</p>
<p>I've merged it. I think we can close this ticket now, Josh, feel free to re-open if there is something else to be done!</p> Arvados - Bug #9918: keep-balance fails with "Malformed index line" errorhttps://dev.arvados.org/issues/9918?journal_id=647442018-07-23T18:52:27ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Release</strong> set to <i>13</i></li></ul>