Actions
Feature #12900
open[Crunch2] [crunch-run] Prune old images before installing image for current container
Story points:
3.0
Release:
Release relationship:
Auto
Description
Before downloading & installing the image for the current container, crunch-run should:
- Check current usage (size of docker images already present on the worker)
- Check whether the image for the current container is already installed; if not, add the estimated image size to the current usage figure
- Check whether any other crunch-run processes are running (if so, skip cleanup to avoid races where an image is deleted between "download" and "start" -- or handle the race in some other way)
- Delete old images as needed to ensure current usage ≤ maximum
- Don't delete images needed for the current container
- Remove docker-cleaner from the install docs
- Add "uninstall docker-cleaner" to the upgrade notes
- Remove docker-cleaner parts from package build/test scripts
- Remove docker-cleaner from the source tree
Ops can temporarily disable cleanup by putting the worker in "hold" state: new containers won't be scheduled, so cleanup won't run.
Related issues
Updated by Tom Morris almost 7 years ago
- Subject changed from [CRUNCH] crunch-run should be able to control the clenaup of container to [Crunch2] Add container cleanup to crunch run
- Target version set to To Be Groomed
Updated by Tom Clegg over 5 years ago
It might work better to clean up before downloading the current image, rather than after the container exits:
- if crunch-run gets killed before it has a chance to clean up, that doesn't affect the prospects of future containers
- we have more information about how much space we need to make for the next (current) container's image
- we can avoid deleting image(s) that will be immediately re-downloaded anyway because they're needed by the next/current container
- arvados-dispatch-cloud's "hold instance" API ("don't shutdown or run more containers") makes the "don't cleanup after" flag unnecessary
Updated by Tom Morris over 5 years ago
- Story points set to 3.0
Story includes getting rid of docker cleaner component and updating documentation.
Updated by Tom Morris over 5 years ago
- Target version changed from To Be Groomed to Arvados Future Sprints
Updated by Tom Clegg over 5 years ago
- Related to Idea #13908: [Epic] Replace SLURM for cloud job scheduling/dispatching added
Updated by Tom Clegg over 5 years ago
- Subject changed from [Crunch2] Add container cleanup to crunch run to [Crunch2] [crunch-run] Prune old images before installing image for current container
- Description updated (diff)
Updated by Tom Clegg about 5 years ago
- Related to Idea #15759: [arvados-dispatch-cloud] deploy/run correct version of crunch-run binary on worker nodes added
Updated by Peter Amstutz over 3 years ago
- Target version deleted (
Arvados Future Sprints)
Updated by Peter Amstutz almost 3 years ago
- Related to Bug #18486: Docker containers are always removed added
Actions