https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-08-18T15:15:33ZArvadosArvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=963112021-08-18T15:15:33ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=966142021-09-01T13:36:40ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-09-01 sprint</i> to <i>2021-09-15 sprint</i></li></ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968032021-09-06T14:31:19ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968492021-09-07T21:24:08ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="17989: Reduces the manifest size limit so that it doesn't fail on extreme cases Arvados-DCO-1.1-..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/ca109d685ba269ec82a433041cdcad6d229950c9">ca109d685</a> - branch <code>17989-pysdk-timeout</code><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2680/"<a href="https://ci.arvados.org/job/developer-run-tests/2680/">developer-run-tests: #2680 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2680" alt="" /></a></a></p>
<ul>
<li>Fixes the <code>arvados.api_from_config()</code> call so that it uses the specified <code>timeout</code> param.</li>
<li>Adds test.</li>
<li>As a bonus, lowers the max manifest size limit on the test collection creation script to avoid "payload is too large" errors.</li>
</ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968562021-09-08T13:45:07ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Updates at <a class="changeset" title="17989: Reduces the manifest size limit so that it doesn't fail on extreme cases Arvados-DCO-1.1-..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/ca109d685ba269ec82a433041cdcad6d229950c9">ca109d685</a> - branch <code>17989-pysdk-timeout</code><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2680/"<a href="https://ci.arvados.org/job/developer-run-tests/2680/">developer-run-tests: #2680 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2680" alt="" /></a></a></p>
<ul>
<li>Fixes the <code>arvados.api_from_config()</code> call so that it uses the specified <code>timeout</code> param.</li>
<li>Adds test.</li>
<li>As a bonus, lowers the max manifest size limit on the test collection creation script to avoid "payload is too large" errors.</li>
</ul>
</blockquote>
<p>That was a unfortunate oversight, but it looks like it was easy to find and fix.</p>
<p>On the subject of timeouts:</p>
<p>We already have feedback that a default timeout 10s is too short (e.g. arv-mount reportedly cannot load very large collections).</p>
<p>Timeouts are one of those things that don't matter until you start hitting them and you actually do need it to wait longer, then they matter a lot.</p>
<p>So I have a couple of follow up suggestions:</p>
<ul>
<li>Raise the timeout to 20 or 30 seconds (let's see what Tom and Ward think)</li>
<li>increase the default timeout of arv-mount, and/or add a command line flag that lets you adjust the timeout and/or add an enviroment variable ARVADOS_TIMEOUT or something that lets you adjust the default timeout.</li>
</ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968572021-09-08T13:46:10ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>I should have said, the 17989-pysdk-timeout branch LGTM.</p> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968592021-09-08T13:53:14ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" href="/issues/17171">Bug #17171</a>: [sdk] unreasonable python SDK timeout when ARVADOS_API_HOST is reachable</i> added</li></ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968602021-09-08T13:54:08ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>(copied from slack)</p>
<p>I don't suppose we have any better granularity?</p>
<p>Initial TCP connection/TLS handshake timeout and subsequent response timeout really ought to be different things.</p>
<p>This is one of the reasons we use pycurl for keepclient, because it gave us a more knobs to control exactly when to abandon a connection.</p> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968682021-09-08T18:19:16ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>From discussion: let's revert the timeout to 300 seconds and investigate further if we can separate the connect/handshake timeout from the response timeout.</p> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968692021-09-08T18:22:06ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="17989: Reverts default PySDK timeout to 5 minutes. Updates test. Arvados-DCO-1.1-Signed-off-by: ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/aab5659260ffe9184e80d09638e472800ab9d128">aab565926</a> - branch <code>17989-revert-pysdk-default-timeout</code><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2681/"<a href="https://ci.arvados.org/job/developer-run-tests/2681/">developer-run-tests: #2681 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2681" alt="" /></a></a></p>
<ul>
<li>Reverts <code>arvados.api()</code>'s default timeout to 300 seconds.</li>
<li>Updates test.</li>
</ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968702021-09-08T18:42:05ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Updates at <a class="changeset" title="17989: Reverts default PySDK timeout to 5 minutes. Updates test. Arvados-DCO-1.1-Signed-off-by: ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/aab5659260ffe9184e80d09638e472800ab9d128">aab565926</a> - branch <code>17989-revert-pysdk-default-timeout</code><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2681/"<a href="https://ci.arvados.org/job/developer-run-tests/2681/">developer-run-tests: #2681 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2681" alt="" /></a></a></p>
<ul>
<li>Reverts <code>arvados.api()</code>'s default timeout to 300 seconds.</li>
<li>Updates test.</li>
</ul>
</blockquote>
<p>LGTM</p> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=968732021-09-08T20:09:17ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset arvados-private:commit:arvados|834f7c726f727e04250ed6b3eda287ca996e9c38.</p> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=971542021-09-20T13:50:10ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>43</i></li></ul> Arvados - Bug #17989: PySDK socket read timeout on big collection create callshttps://dev.arvados.org/issues/17989?journal_id=978952021-10-22T18:29:08ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Release</strong> changed from <i>43</i> to <i>42</i></li></ul>