Feature #17813
Updated by Tom Clegg almost 3 years ago
Currently, when crunch-run is in singularity mode, it converts the specified docker image to a .sif file before starting the container. The conversion is slow, and the .sif file is not saved anywhere, so the entire process is slow even when the same image is used for lots of steps in a workflow, sequential containers running on the same worker node, etc.
We can speed this up in most cases by saving the .sif file in Keep.
* Before converting the docker image, check a project called "auto-generated singularity images" in a ".cache" project in the home project (runtime_user_uuid) for a collection named "singularity image for X" (where X is the docker image PDH) containing a single .sif file. If that exists, extend the trash_at time to now+2w, and use that .sif file instead of converting the docker image.
* If that doesn't work out, convert the docker image as we do now.
* After converting, save the .sif file in "singularity image for X" with trash_at=now+2w and properties {"type":"singularity image","docker_image_pdh":X}. image","docker_image_pdh":"..."}.