Scoping/grooming Singularity support work
Need to support different image types: docker vs singularity
? Do whatever the Kubernetes community is doing for storing images / downloading / managing images ?
#3 Updated by Ward Vandewege 9 months ago
docker hub => singularity hub
singularity registry server (https://singularityhub.github.io/sregistry): open source version of singularity hub, for our own deployment. Python though, which is problematic.
singularity user guide https://sylabs.io/guides/3.7/user-guide/
The singularity tool can manage oci containers but it's a separate command 'singularity oci' vs 'singularity instance'
Singularity Image Format (SIF) seems to be it's own thing. It's different from the OCI image format. The singularity command can convert OCI images into SIF images.
Singularity can run docker images directly, it seems:
singularity run docker://arvados/jobs
Can SIF images be converted back to docker images?
Is Podman something we should explore?
K8s is using registries (local or not, cf. https://kubernetes.io/docs/concepts/containers/images/), and oci image manifests (https://github.com/opencontainers/image-spec/blob/master/manifest.md). And they are using containerd now, instead of docker.
Maybe we can make a keep-backed container registry. Our existing docker image tarballs could hopefully be translated/exposed on the fly that way.
Links to explore:
https://github.com/moby/moby/issues/1266 (mandatory compression)
Does singularity support the namespace/process limits we use in docker? E.g. the ram limits. Are there kernel version requirements?
cwltool already supports 4 different container runners, including podman. It translates images on the fly. Either docker-compatible registry, or local images.
docker images have a name + digest. It would be cool to move away from the current system where we use links in arvados to look up the correct collection.
It looks like https://github.com/containers/skopeo can load 'docker save' images (like the ones we have in Keep) and convert them to OCI images. Interestingly, https://umo.ci/quick-start/getting-an-image/ says that while umoci can do whatever you want with OCI images, the way to get OCI images is to convert them from another format. For now.
#8 Updated by Ward Vandewege 9 months ago
Singularity can convert docker tar files aka the output of 'docker save', which is what we store in Keep with arv-keepdocker into a Singularity Image File (SIF) and run it, without requiring root:
$ docker pull arvados/jobs:latest latest: Pulling from arvados/jobs 8559a31e96f4: Already exists 017576171510: Pull complete 8b47f0d30222: Pull complete c73cfde5c474: Pull complete 67c8c916cf1a: Pull complete be6affc9cc07: Pull complete da4b1effc0b3: Pull complete db23a5ceaaee: Pull complete 6a212e3ef088: Pull complete 63feb2687d75: Pull complete 0c0175320747: Pull complete Digest: sha256:5ad1019f39bb3284c56e2c084455b582f9a2c2899e56891efdbcedf360a9120e Status: Downloaded newer image for arvados/jobs:latest docker.io/arvados/jobs:latest $ docker save arvados/jobs:latest > arvados-jobs.latest.tar $ ls -laF arvados-jobs.latest.tar -rw-r--r-- 1 ward ward 295209984 Jan 14 17:16 arvados-jobs.latest.tar $ singularity build arvados-jobs.latest.sif docker-archive://arvados-jobs.latest.tar INFO: Starting build... Getting image source signatures Copying blob 13cb14c2acd3 done Copying blob 9c4f3c839d9c done Copying blob 8a37eeecff8a done Copying blob 5b7fe323e31d done Copying blob ef2b3b5d5b5e done Copying blob 6d743f7687b8 done Copying blob bd000de1bb10 done Copying blob bc032c9eff1c done Copying blob bf7bd6469707 done Copying blob d5276026d90b done Copying blob d64677006613 done Copying config 12654f1454 done Writing manifest to image destination Storing signatures 2021/01/14 17:16:54 info unpack layer: sha256:bc476b6201e817669671caaeaa024328d26aa7af2d3223cb2fcf9ab1947c5db6 2021/01/14 17:16:55 info unpack layer: sha256:80afac547c42c232a4dd82a2fbd0e7ddc1b4b42c6d0417f7b3dfa8d907a675ac 2021/01/14 17:16:55 info unpack layer: sha256:567d5834c5d25915ffeda7095aeffac5232db5be514b3aba3daa822b734a78d8 2021/01/14 17:16:55 info unpack layer: sha256:4c7322e5ac9f2176a83da85ac0004cfe2bfd5287ab9f6568d942d6bd29bac768 2021/01/14 17:16:55 info unpack layer: sha256:b46dfe7683cdaba75e319f74fa6f61812ab519bc2ea72456eefcceaab1a4c06f 2021/01/14 17:16:55 info unpack layer: sha256:4a13af3199c14391d66ac30e6ad9edd6c2a2b07e292ee72ccd2ef895333e28a1 2021/01/14 17:16:55 info unpack layer: sha256:38a3d47ebbe5ee33971cba0a4131ae031767ccf2f0200d51d70fb108f07ecf7e 2021/01/14 17:16:55 info unpack layer: sha256:e7607c06e3ab3e2b91e8215b63c3080a0627b1dbab26712e3a3791ffe59c97d1 2021/01/14 17:16:57 info unpack layer: sha256:ef48bdff5f54fd5db7eacc81872f095923db3df5b69c3bac8c59bf5711193ec1 2021/01/14 17:16:57 info unpack layer: sha256:c98648c6dc5e486c966f7841f1582c105519911c7fb1d500e372d418807303be 2021/01/14 17:16:57 info unpack layer: sha256:d1e3135d5d225564693f94cafb611935756c98f598d04db1b9185c112fb4e113 INFO: Creating SIF file... INFO: Build complete: arvados-jobs.latest.sif $ singularity run arvados-jobs.latest.sif bash: ps: command not found Singularity>
The singularity user manual has a number of best practices on running docker images: https://sylabs.io/guides/3.7/user-guide/singularity_and_docker.html#best-practices. They point out the big difference between singularity and docker/containerd: singularity runs containers in userspace. No root required, which is the big attraction. This comes with a number of consequences, which may affect real world container use...
There is a `singularity capability` command that can be used to add capabilities to a certain user or group:
$ singularity capability Error for command "capability": Invalid command Usage: singularity [global options...] capability <command> Available Commands: add Add capabilities to a user or group (requires root) avail Show description for available capabilities drop Remove capabilities from a user or group (requires root) list Show capabilities for a given user or group Run 'singularity --help' for more detailed usage information.
Singularity can also run oci images directly, but all those commands require root, which defeats the purpose:
$ singularity oci Manage OCI containers Usage: singularity oci Description: Allow you to manage containers from OCI bundle directories. NOTE: all oci commands requires to run as root Options: -h, --help help for oci Available Commands: attach Attach console to a running container process (root user only) create Create a container from a bundle directory (root user only) delete Delete container (root user only) exec Execute a command within container (root user only) kill Kill a container (root user only) mount Mount create an OCI bundle from SIF image (root user only) pause Suspends all processes inside the container (root user only) resume Resumes all processes previously paused inside the container (root user only) run Create/start/attach/delete a container from a bundle directory (root user only) start Start container process (root user only) state Query state of a container (root user only) umount Umount delete bundle (root user only) update Update container cgroups resources (root user only) Examples: All group commands have their own help output: $ singularity oci create -b ~/bundle mycontainer $ singularity oci start mycontainer For additional help or support, please visit https://www.sylabs.io/docs/
Singularity can also run docker images natively, from dockerhub or a local repository, e.g.
singularity run docker://arvados/jobs
(there is also support for authentication)
#10 Updated by Ward Vandewege 9 months ago
Starting assumption: we still use docker images, use arv-keepdocker, etc. None of that changes.
Perhaps there is a bare bones MVP of singularity support that looks a little like this.
When we run a container on a compute node, we do a container conversion, on the fly, to a SIF file, and run that with singularity instead. Perhaps we even save the SIF file in Keep and do something with another Link object to make it findable in the future, for the corresponding docker image. TODO: check if the framework we built in for the docker image format v1 -> v2 could be used here.
It takes a bit of time to do the conversion to SIF, and big images could consume a considerable amount of disk space, which may be a problem on a compute node. There will probably be some capabilities that will need to be set (by root, as part of creating the compute image/as part of the HPC env) for the user that will run the singularity containers.
We add a docker repository interface to Keep, which exposes our stored images. Then run those images with the native docker support in singularity.