https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422016-11-24T10:17:09ZArvadosArvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=457672016-11-24T10:17:09ZJoshua Randalljr17@sanger.ac.uk
<ul></ul><p>I ran into pretty much exactly these two error messages (from the same job) after upgrading systemd to the latest version (v230 from jessie-backports in my case), which appears to have issues with docker. The underlying problem seems to be that the system.slice directory is no longer present in that version.</p>
<p>The workaround was to switch docker to not use systemd for managing cgroups: <a class="external" href="https://github.com/docker/docker/issues/17653#issuecomment-155609224">https://github.com/docker/docker/issues/17653#issuecomment-155609224</a></p>
<p>If the fix for this issue obscured the error messages that come from docker, I'd never have figured out the real problem, so whatever the fix is here should probably make sure the errors are (also) logged.</p> Arvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=457712016-11-26T14:18:57ZWard Vandewegeward@curii.com
<ul></ul><p>Joshua Randall wrote:</p>
<blockquote>
<p>I ran into pretty much exactly these two error messages (from the same job) after upgrading systemd to the latest version (v230 from jessie-backports in my case), which appears to have issues with docker. The underlying problem seems to be that the system.slice directory is no longer present in that version.</p>
<p>The workaround was to switch docker to not use systemd for managing cgroups: <a class="external" href="https://github.com/docker/docker/issues/17653#issuecomment-155609224">https://github.com/docker/docker/issues/17653#issuecomment-155609224</a></p>
<p>If the fix for this issue obscured the error messages that come from docker, I'd never have figured out the real problem, so whatever the fix is here should probably make sure the errors are (also) logged.</p>
</blockquote>
<p>I agree - note that this ticket was a bit out of date - I also figured out a couple weeks ago that the other failure that can lead to this error is the cgroup thing you identified. Interpreting errors could be useful to give users a hint of what may be going on, but we shouldn't obscure the underlying errors.</p> Arvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=545252017-08-29T13:48:55ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=947742021-07-07T18:23:05ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> deleted (<del><i>Arvados Future Sprints</i></del>)</li></ul> Arvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=1120222023-02-14T22:23:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>60</i></li></ul> Arvados - Bug #10182: Provide more reasonable error messages for memory issues during container dispatchhttps://dev.arvados.org/issues/10182?journal_id=1233932024-03-01T21:13:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>Future</i></li></ul>