https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422015-10-02T14:35:26ZArvadosArvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=309382015-10-02T14:35:26ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Subject</strong> changed from <i>[Crunch] Job containers not removed consistently, filling disk</i> to <i>[Crunch] Docker container not removed when job canceled, filling disk</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/30938/diff?detail_id=30376">diff</a>)</li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=310472015-10-06T18:25:26ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/31047/diff?detail_id=30477">diff</a>)</li><li><strong>Story points</strong> set to <i>2.0</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=312342015-10-13T18:28:59ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-10-28 sprint</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=313022015-10-14T19:18:54ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Peter Amstutz</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=314612015-10-16T18:19:34ZBrett Smithbrett.smith@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-10-28 sprint</i> to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=319302015-10-28T19:51:54ZTom Cleggtom@curii.com
<ul><li><strong>Assigned To</strong> changed from <i>Peter Amstutz</i> to <i>Tom Clegg</i></li><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-11-11 sprint</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320092015-10-30T22:25:41ZTom Cleggtom@curii.com
<ul></ul><p>Naming containers sounds like a good idea anyway, but seems tangential. Unless dockercleaner is supposed to pay attention to the names, perhaps in order to exempt non-Crunch containers from automatic removal...?</p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320142015-10-30T22:48:37ZTom Cleggtom@curii.com
<ul></ul><p>Should dockercleaner also delete all stopped containers that are already present when it starts up? This would help keep a long-running (e.g., bare metal) worker node clean.</p>
If/when we do add this, I think it should have a separate command line flag, to support a workflow like
<ol>
<li>Turn off dockercleaner</li>
<li>Run a job</li>
<li>Turn on dockercleaner <code>--leave-existing-containers</code></li>
<li>Inspect the container left behind by the above job, but let subsequent jobs get cleaned up</li>
</ol>
<p>Until then, there's <code>"docker ps --filter status=exited --format {{.ID}} | xargs docker rm"</code>.</p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320152015-10-30T22:50:02ZTom Cleggtom@curii.com
<ul></ul><p>7444-dockercleaner-containers @ <a class="changeset" title="7444: Note automatic removal of stopped containers, and how to disable." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/e10ccaba824b4f60ddc516903304351496b5fdca">e10ccab</a></p>
<p>7444-no-docker-rm @ <a class="changeset" title="7444: Do not remove docker containers with docker --rm; let dockercleaner do it." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/07beca79dfa589fe35dad751f6a25a6ce43cefe4">07beca7</a></p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320162015-10-31T00:24:03ZBrett Smithbrett.smith@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>Naming containers sounds like a good idea anyway, but seems tangential.</p>
</blockquote>
<p>You are right it is not necessary for the dockercleaner changes. I previously had an implementation idea based on naming containers predictably and having crunch-job remove them. This is basically a remnant of that—there was still interest in naming as a debugging aid.</p>
<blockquote>
<p>Should dockercleaner also delete all stopped containers that are present when it starts up?</p>
</blockquote>
<p>I'm interested in ops' opinion on this but my vote is yes.</p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320372015-11-02T20:54:46ZTom Cleggtom@curii.com
<ul></ul><p>Both changes (dockercleaner and crunch-job) are now in 7444-dockercleaner-containers at <a class="changeset" title="7444: Rename kwarg remove_stopped_containers -> remove_containers_onexit" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9b48b17eddea5e366e0c59ed9f3540793550256c">9b48b17</a></p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320382015-11-02T20:54:53ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320592015-11-03T16:28:59ZNico César
<ul></ul><p>review 9b48b17eddea5e366e0c59ed9f3540793550256c</p>
<p>LGTM</p> Arvados - Bug #7444: [Crunch] Docker container not removed when job canceled, filling diskhttps://dev.arvados.org/issues/7444?journal_id=320892015-11-04T06:30:10ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>80</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:1d1c6de3c842a33a57b7d469fdaaaa1b873433dc.</p>