Feature #12900

[Crunch2] [crunch-run] Prune old images before installing image for current container

Added by Nico César over 4 years ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
3.0

Description

Before downloading & installing the image for the current container, crunch-run should:
  • Check current usage (size of docker images already present on the worker)
  • Check whether the image for the current container is already installed; if not, add the estimated image size to the current usage figure
  • Check whether any other crunch-run processes are running (if so, skip cleanup to avoid races where an image is deleted between "download" and "start" -- or handle the race in some other way)
  • Delete old images as needed to ensure current usage ≤ maximum
  • Don't delete images needed for the current container
This feature eliminates the need for the docker-cleaner service, so we also need to
  • Remove docker-cleaner from the install docs
  • Add "uninstall docker-cleaner" to the upgrade notes
  • Remove docker-cleaner parts from package build/test scripts
  • Remove docker-cleaner from the source tree

Ops can temporarily disable cleanup by putting the worker in "hold" state: new containers won't be scheduled, so cleanup won't run.


Related issues

Related to Arvados - Story #13908: [Epic] Replace SLURM for cloud job scheduling/dispatchingResolved

Related to Arvados - Story #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodesResolved12/30/2019

Related to Arvados - Bug #18486: Docker containers are always removedNew

History

#2 Updated by Nico César over 4 years ago

  • Description updated (diff)

#3 Updated by Tom Morris over 4 years ago

  • Subject changed from [CRUNCH] crunch-run should be able to control the clenaup of container to [Crunch2] Add container cleanup to crunch run
  • Target version set to To Be Groomed

#4 Updated by Peter Amstutz about 3 years ago

  • Tracker changed from Bug to Feature

#5 Updated by Tom Clegg about 3 years ago

It might work better to clean up before downloading the current image, rather than after the container exits:
  • if crunch-run gets killed before it has a chance to clean up, that doesn't affect the prospects of future containers
  • we have more information about how much space we need to make for the next (current) container's image
  • we can avoid deleting image(s) that will be immediately re-downloaded anyway because they're needed by the next/current container
  • arvados-dispatch-cloud's "hold instance" API ("don't shutdown or run more containers") makes the "don't cleanup after" flag unnecessary

#6 Updated by Tom Morris about 3 years ago

  • Story points set to 3.0

Story includes getting rid of docker cleaner component and updating documentation.

#7 Updated by Tom Morris about 3 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

#8 Updated by Tom Clegg almost 3 years ago

  • Related to Story #13908: [Epic] Replace SLURM for cloud job scheduling/dispatching added

#9 Updated by Tom Clegg almost 3 years ago

  • Subject changed from [Crunch2] Add container cleanup to crunch run to [Crunch2] [crunch-run] Prune old images before installing image for current container
  • Description updated (diff)

#10 Updated by Tom Clegg over 2 years ago

  • Related to Story #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodes added

#11 Updated by Peter Amstutz 11 months ago

  • Target version deleted (Arvados Future Sprints)

#12 Updated by Peter Amstutz 5 months ago

  • Project changed from Arvados Private to Arvados

#13 Updated by Peter Amstutz 5 months ago

  • Related to Bug #18486: Docker containers are always removed added

Also available in: Atom PDF