Bug #18258
Updated by Peter Amstutz over 2 years ago
When singularity converts a docker container, it has to extract and stash the layers while it is working. It keeps them around in case it needs them again. It uses ~/.singularity/cache or $SINGULARITY_CACHEDIR. On a static node, these layers can accumulate over time and fill up the disk, so we should clean them up when we don't need them any more.
Singularity also uses scratch space in $TMPDIR (and/or $SINGULARITY_TMPDIR).
Propose that crunch-run should set SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a temporary directory on the host's data partition (this is the partition that will also be used for the container's scratch space and output). After the image is converted, it should delete the temporary singularity directory.
reference: https://sylabs.io/guides/3.7//user-guide/build_env.html#sec-cache