Project

General

Profile

Actions
Copy link

Bug #18258

closed

Manage Singularity download cache

Added by Peter Amstutz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
2021-10-13 sprint
Story points:
-
Release:
Arvados 2.3.0
Release relationship:
Auto

Description

When singularity converts a docker container, it has to extract and stash the layers while it is working. It keeps them around in case it needs them again. It uses ~/.singularity/cache or $SINGULARITY_CACHEDIR. On a static node, these layers can accumulate over time and fill up the disk, so we should clean them up when we don't need them any more.

Singularity also uses scratch space in $TMPDIR (and/or $SINGULARITY_TMPDIR).

Propose that crunch-run should set SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a temporary directory on the host's data partition (this is the partition that will also be used for the container's scratch space and output). After the image is converted, it should run singularity cache clean -f and then delete the temporary singularity directory.

reference: https://sylabs.io/guides/3.7//user-guide/build_env.html#sec-cache

Actions
Copy link

Also available in: Atom PDF