Idea #17296
Updated by Peter Amstutz almost 4 years ago
When we run a container on a compute node, we do a container conversion, on the fly, to a SIF file, and run that with singularity instead. Perhaps we even save the SIF file in Keep and do something with another Link object to make it findable in the future, for the corresponding docker image. TODO: check if the framework we built in for the docker image format v1 -> v2 could be used here.
# global option that switches between docker or singularity runner
# container_request runtime parameters flag that chooses between docker and singularity
# crunch-run gets docker tar file from keep (existing docker v2 format images)
# crunch-run converts docker tar file to SIF:
<pre>
$ docker save arvados/jobs:latest > arvados-jobs.latest.tar
$ ls -laF arvados-jobs.latest.tar
-rw-r--r-- 1 ward ward 295209984 Jan 14 17:16 arvados-jobs.latest.tar
$ singularity build arvados-jobs.latest.sif docker-archive://arvados-jobs.latest.tar
INFO: Starting build...
...
</pre>
# crunch-run executes singularity with mount points, stdout/stderr captured to logs, etc
# slurm dispatcher supports singularity
## ideally the backend container runner should be transparent to the dispatcher
# proof of concept will be tested on 9tee4
# assume that user id inside the container will be the same as the crunch-run user (?)