Project

General

Profile

Actions

Feature #12900

open

[Crunch2] [crunch-run] Prune old images before installing image for current container

Added by Nico César over 6 years ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Story points:
3.0
Release:
Release relationship:
Auto

Description

Before downloading & installing the image for the current container, crunch-run should:
  • Check current usage (size of docker images already present on the worker)
  • Check whether the image for the current container is already installed; if not, add the estimated image size to the current usage figure
  • Check whether any other crunch-run processes are running (if so, skip cleanup to avoid races where an image is deleted between "download" and "start" -- or handle the race in some other way)
  • Delete old images as needed to ensure current usage ≤ maximum
  • Don't delete images needed for the current container
This feature eliminates the need for the docker-cleaner service, so we also need to
  • Remove docker-cleaner from the install docs
  • Add "uninstall docker-cleaner" to the upgrade notes
  • Remove docker-cleaner parts from package build/test scripts
  • Remove docker-cleaner from the source tree

Ops can temporarily disable cleanup by putting the worker in "hold" state: new containers won't be scheduled, so cleanup won't run.


Related issues

Related to Arvados - Idea #13908: [Epic] Replace SLURM for cloud job scheduling/dispatchingResolvedActions
Related to Arvados - Idea #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodesResolvedTom Clegg12/30/2019Actions
Related to Arvados - Bug #18486: Docker containers are always removedNewActions
Actions #2

Updated by Nico César over 6 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Morris over 6 years ago

  • Subject changed from [CRUNCH] crunch-run should be able to control the clenaup of container to [Crunch2] Add container cleanup to crunch run
  • Target version set to To Be Groomed
Actions #4

Updated by Peter Amstutz about 5 years ago

  • Tracker changed from Bug to Feature
Actions #5

Updated by Tom Clegg about 5 years ago

It might work better to clean up before downloading the current image, rather than after the container exits:
  • if crunch-run gets killed before it has a chance to clean up, that doesn't affect the prospects of future containers
  • we have more information about how much space we need to make for the next (current) container's image
  • we can avoid deleting image(s) that will be immediately re-downloaded anyway because they're needed by the next/current container
  • arvados-dispatch-cloud's "hold instance" API ("don't shutdown or run more containers") makes the "don't cleanup after" flag unnecessary
Actions #6

Updated by Tom Morris about 5 years ago

  • Story points set to 3.0

Story includes getting rid of docker cleaner component and updating documentation.

Actions #7

Updated by Tom Morris about 5 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
Actions #8

Updated by Tom Clegg almost 5 years ago

  • Related to Idea #13908: [Epic] Replace SLURM for cloud job scheduling/dispatching added
Actions #9

Updated by Tom Clegg almost 5 years ago

  • Subject changed from [Crunch2] Add container cleanup to crunch run to [Crunch2] [crunch-run] Prune old images before installing image for current container
  • Description updated (diff)
Actions #10

Updated by Tom Clegg over 4 years ago

  • Related to Idea #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodes added
Actions #11

Updated by Peter Amstutz almost 3 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #12

Updated by Peter Amstutz over 2 years ago

  • Project changed from 35 to Arvados
Actions #13

Updated by Peter Amstutz over 2 years ago

  • Related to Bug #18486: Docker containers are always removed added
Actions #14

Updated by Peter Amstutz about 1 year ago

  • Release set to 60
Actions #15

Updated by Peter Amstutz about 2 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF