Arvados

docker hub => singularity hub

singularity registry server (https://singularityhub.github.io/sregistry): open source version of singularity hub, for our own deployment. Python though, which is problematic.

singularity user guide https://sylabs.io/guides/3.7/user-guide/

The singularity tool can manage oci containers but it's a separate command 'singularity oci' vs 'singularity instance'

Singularity Image Format (SIF) seems to be it's own thing. It's different from the OCI image format. The singularity command can convert OCI images into SIF images.

Singularity can run docker images directly, it seems:

singularity run docker://arvados/jobs

Can SIF images be converted back to docker images?

Is Podman something we should explore?

K8s is using registries (local or not, cf. https://kubernetes.io/docs/concepts/containers/images/), and oci image manifests (https://github.com/opencontainers/image-spec/blob/master/manifest.md). And they are using containerd now, instead of docker.

Maybe we can make a keep-backed container registry. Our existing docker image tarballs could hopefully be translated/exposed on the fly that way.

Links to explore:
https://github.com/opencontainers/image-spec/blob/v1.0.1/config.md#explanation
https://github.com/opencontainers/distribution-spec/blob/master/spec.md#endpoints
https://github.com/moby/moby/issues/1266 (mandatory compression)
https://docs.docker.com/registry/spec/api/

Does singularity support the namespace/process limits we use in docker? E.g. the ram limits. Are there kernel version requirements?

cwltool already supports 4 different container runners, including podman. It translates images on the fly. Either docker-compatible registry, or local images.

docker images have a name + digest. It would be cool to move away from the current system where we use links in arvados to look up the correct collection.

It looks like https://github.com/containers/skopeo can load 'docker save' images (like the ones we have in Keep) and convert them to OCI images. Interestingly, https://umo.ci/quick-start/getting-an-image/ says that while umoci can do whatever you want with OCI images, the way to get OCI images is to convert them from another format. For now.

Few questions

Can we use OCI containers

Can we use tools like umoci

If Kubernetes can use singularity, how does k8s storing and distributing singularity images.

I would also be very happy if we could come up with a new way to handle container images in general because the current scheme of link objects pointing to keep collections has a bunch of issues.

How does singularity handle networking?

How do you build images?

Singularity can convert docker tar files aka the output of 'docker save', which is what we store in Keep with arv-keepdocker into a Singularity Image File (SIF) and run it, without requiring root:

$ docker pull arvados/jobs:latest
latest: Pulling from arvados/jobs
8559a31e96f4: Already exists 
017576171510: Pull complete 
8b47f0d30222: Pull complete 
c73cfde5c474: Pull complete 
67c8c916cf1a: Pull complete 
be6affc9cc07: Pull complete 
da4b1effc0b3: Pull complete 
db23a5ceaaee: Pull complete 
6a212e3ef088: Pull complete 
63feb2687d75: Pull complete 
0c0175320747: Pull complete 
Digest: sha256:5ad1019f39bb3284c56e2c084455b582f9a2c2899e56891efdbcedf360a9120e
Status: Downloaded newer image for arvados/jobs:latest
docker.io/arvados/jobs:latest

$ docker save arvados/jobs:latest > arvados-jobs.latest.tar

$ ls -laF arvados-jobs.latest.tar 
-rw-r--r-- 1 ward ward 295209984 Jan 14 17:16 arvados-jobs.latest.tar

$ singularity build arvados-jobs.latest.sif docker-archive://arvados-jobs.latest.tar
INFO:    Starting build...
Getting image source signatures
Copying blob 13cb14c2acd3 done  
Copying blob 9c4f3c839d9c done  
Copying blob 8a37eeecff8a done  
Copying blob 5b7fe323e31d done  
Copying blob ef2b3b5d5b5e done  
Copying blob 6d743f7687b8 done  
Copying blob bd000de1bb10 done  
Copying blob bc032c9eff1c done  
Copying blob bf7bd6469707 done  
Copying blob d5276026d90b done  
Copying blob d64677006613 done  
Copying config 12654f1454 done  
Writing manifest to image destination
Storing signatures
2021/01/14 17:16:54  info unpack layer: sha256:bc476b6201e817669671caaeaa024328d26aa7af2d3223cb2fcf9ab1947c5db6
2021/01/14 17:16:55  info unpack layer: sha256:80afac547c42c232a4dd82a2fbd0e7ddc1b4b42c6d0417f7b3dfa8d907a675ac
2021/01/14 17:16:55  info unpack layer: sha256:567d5834c5d25915ffeda7095aeffac5232db5be514b3aba3daa822b734a78d8
2021/01/14 17:16:55  info unpack layer: sha256:4c7322e5ac9f2176a83da85ac0004cfe2bfd5287ab9f6568d942d6bd29bac768
2021/01/14 17:16:55  info unpack layer: sha256:b46dfe7683cdaba75e319f74fa6f61812ab519bc2ea72456eefcceaab1a4c06f
2021/01/14 17:16:55  info unpack layer: sha256:4a13af3199c14391d66ac30e6ad9edd6c2a2b07e292ee72ccd2ef895333e28a1
2021/01/14 17:16:55  info unpack layer: sha256:38a3d47ebbe5ee33971cba0a4131ae031767ccf2f0200d51d70fb108f07ecf7e
2021/01/14 17:16:55  info unpack layer: sha256:e7607c06e3ab3e2b91e8215b63c3080a0627b1dbab26712e3a3791ffe59c97d1
2021/01/14 17:16:57  info unpack layer: sha256:ef48bdff5f54fd5db7eacc81872f095923db3df5b69c3bac8c59bf5711193ec1
2021/01/14 17:16:57  info unpack layer: sha256:c98648c6dc5e486c966f7841f1582c105519911c7fb1d500e372d418807303be
2021/01/14 17:16:57  info unpack layer: sha256:d1e3135d5d225564693f94cafb611935756c98f598d04db1b9185c112fb4e113
INFO:    Creating SIF file...
INFO:    Build complete: arvados-jobs.latest.sif

$ singularity run arvados-jobs.latest.sif
bash: ps: command not found
Singularity>

The singularity user manual has a number of best practices on running docker images: https://sylabs.io/guides/3.7/user-guide/singularity_and_docker.html#best-practices. They point out the big difference between singularity and docker/containerd: singularity runs containers in userspace. No root required, which is the big attraction. This comes with a number of consequences, which may affect real world container use...

There is a `singularity capability` command that can be used to add capabilities to a certain user or group:

$ singularity capability
Error for command "capability": Invalid command

Usage:
  singularity [global options...] capability <command>

Available Commands:
  add         Add capabilities to a user or group (requires root)
  avail       Show description for available capabilities
  drop        Remove capabilities from a user or group (requires root)
  list        Show capabilities for a given user or group

Run 'singularity --help' for more detailed usage information.

Singularity can also run oci images directly, but all those commands require root, which defeats the purpose:

$ singularity oci
Manage OCI containers

Usage:
  singularity oci

Description:
  Allow you to manage containers from OCI bundle directories.

  NOTE: all oci commands requires to run as root

Options:
  -h, --help   help for oci

Available Commands:
  attach      Attach console to a running container process (root user only)
  create      Create a container from a bundle directory (root user only)
  delete      Delete container (root user only)
  exec        Execute a command within container (root user only)
  kill        Kill a container (root user only)
  mount       Mount create an OCI bundle from SIF image (root user only)
  pause       Suspends all processes inside the container (root user only)
  resume      Resumes all processes previously paused inside the container (root user only)
  run         Create/start/attach/delete a container from a bundle directory (root user only)
  start       Start container process (root user only)
  state       Query state of a container (root user only)
  umount      Umount delete bundle (root user only)
  update      Update container cgroups resources (root user only)

Examples:
  All group commands have their own help output:

  $ singularity oci create -b ~/bundle mycontainer
  $ singularity oci start mycontainer

For additional help or support, please visit https://www.sylabs.io/docs/

Singularity can also run docker images natively, from dockerhub or a local repository, e.g.

singularity run docker://arvados/jobs

(there is also support for authentication)

Starting assumption: we still use docker images, use arv-keepdocker, etc. None of that changes.

Perhaps there is a bare bones MVP of singularity support that looks a little like this.

Option 1:

When we run a container on a compute node, we do a container conversion, on the fly, to a SIF file, and run that with singularity instead. Perhaps we even save the SIF file in Keep and do something with another Link object to make it findable in the future, for the corresponding docker image. TODO: check if the framework we built in for the docker image format v1 -> v2 could be used here.

It takes a bit of time to do the conversion to SIF, and big images could consume a considerable amount of disk space, which may be a problem on a compute node. There will probably be some capabilities that will need to be set (by root, as part of creating the compute image/as part of the HPC env) for the user that will run the singularity containers.

Option 2:

We add a docker repository interface to Keep, which exposes our stored images. Then run those images with the native docker support in singularity.