Bug #18258: Manage Singularity download cache - Arvados

Actions

Copy link

Bug #18258

closed

Manage Singularity download cache

Added by Peter Amstutz over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Category:

Crunch

Target version:

2021-10-13 sprint

Story points:

Release:

Arvados 2.3.0

Release relationship:

Auto

Description

When singularity converts a docker container, it has to extract and stash the layers while it is working. It keeps them around in case it needs them again. It uses ~/.singularity/cache or $SINGULARITY_CACHEDIR. On a static node, these layers can accumulate over time and fill up the disk, so we should clean them up when we don't need them any more.

Singularity also uses scratch space in $TMPDIR (and/or $SINGULARITY_TMPDIR).

Propose that crunch-run should set SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a temporary directory on the host's data partition (this is the partition that will also be used for the container's scratch space and output). After the image is converted, it should run singularity cache clean -f and then delete the temporary singularity directory.

reference: https://sylabs.io/guides/3.7//user-guide/build_env.html#sec-cache

Actions

Copy link

Updated by Peter Amstutz over 2 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz over 2 years ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz over 2 years ago

Description updated (diff)

Actions

Copy link

Updated by Ward Vandewege over 2 years ago

Patch at d8ecd9d8940a50ebabe089eb326753f54bb2831b on 18258-set-singularity-cachedir sets SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a path inside the temp/scratch space for the build command, and deletes these directories after. Running `singularity cache clean -f` explicitly first seems superfluous.

Tests are running at developer-run-tests: #2728

Actions

Copy link

Updated by Peter Amstutz over 2 years ago

Ward Vandewege wrote:

Patch at d8ecd9d8940a50ebabe089eb326753f54bb2831b on 18258-set-singularity-cachedir sets SINGULARITY_CACHEDIR and SINGULARITY_TMPDIR to a path inside the temp/scratch space for the build command, and deletes these directories after. Running `singularity cache clean -f` explicitly first seems superfluous.

Tests are running at developer-run-tests: #2728

This LGTM. The reason I suggested using `singularity cache clean -f` was that if it was storing unpacked image layers, you would have files that are owned by root that can't be deleted by the crunch user. But maybe it doesn't do that.

Actions

Copy link