Bug #18258
closed
Manage Singularity download cache
Description
When singularity converts a docker container, it has to extract and stash the layers while it is working. It keeps them around in case it needs them again. It uses ~/.singularity/cache or $SINGULARITY_CACHEDIR. On a static node, these layers can accumulate over time and fill up the disk, so we should clean them up when we don't need them any more.
Singularity also uses scratch space in $TMPDIR (and/or $SINGULARITY_TMPDIR).
Propose that crunch-run should set SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a temporary directory on the host's data partition (this is the partition that will also be used for the container's scratch space and output). After the image is converted, it should run singularity cache clean -f and then delete the temporary singularity directory.
reference: https://sylabs.io/guides/3.7//user-guide/build_env.html#sec-cache
Updated by Peter Amstutz about 3 years ago
- Status changed from New to In Progress
Updated by Ward Vandewege about 3 years ago
Patch at d8ecd9d8940a50ebabe089eb326753f54bb2831b on 18258-set-singularity-cachedir sets SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a path inside the temp/scratch space for the build command, and deletes these directories after. Running `singularity cache clean -f` explicitly first seems superfluous.
Tests are running at developer-run-tests: #2728
Updated by Peter Amstutz about 3 years ago
Ward Vandewege wrote:
Patch at d8ecd9d8940a50ebabe089eb326753f54bb2831b on 18258-set-singularity-cachedir sets SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a path inside the temp/scratch space for the build command, and deletes these directories after. Running `singularity cache clean -f` explicitly first seems superfluous.
Tests are running at developer-run-tests: #2728
This LGTM. The reason I suggested using `singularity cache clean -f` was that if it was storing unpacked image layers, you would have files that are owned by root that can't be deleted by the crunch user. But maybe it doesn't do that.
Updated by Ward Vandewege about 3 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|eb6b9a3d52cab3f7904bae457a9c210e2a76bf6a.