Project

General

Profile

Actions

Feature #12900

open

[Crunch2] [crunch-run] Prune old images before installing image for current container

Added by Nico César over 6 years ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
3.0
Release:
Release relationship:
Auto

Description

Before downloading & installing the image for the current container, crunch-run should:
  • Check current usage (size of docker images already present on the worker)
  • Check whether the image for the current container is already installed; if not, add the estimated image size to the current usage figure
  • Check whether any other crunch-run processes are running (if so, skip cleanup to avoid races where an image is deleted between "download" and "start" -- or handle the race in some other way)
  • Delete old images as needed to ensure current usage ≤ maximum
  • Don't delete images needed for the current container
This feature eliminates the need for the docker-cleaner service, so we also need to
  • Remove docker-cleaner from the install docs
  • Add "uninstall docker-cleaner" to the upgrade notes
  • Remove docker-cleaner parts from package build/test scripts
  • Remove docker-cleaner from the source tree

Ops can temporarily disable cleanup by putting the worker in "hold" state: new containers won't be scheduled, so cleanup won't run.


Related issues

Related to Arvados - Idea #13908: [Epic] Replace SLURM for cloud job scheduling/dispatchingResolvedActions
Related to Arvados - Idea #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodesResolvedTom Clegg12/30/2019Actions
Related to Arvados - Bug #18486: Docker containers are always removedNewActions
Actions

Also available in: Atom PDF