Idea #11724
Updated by Peter Amstutz about 6 years ago
Some users do not have access to Docker on the client. It should be possible to submit a compute node job which can pull (and/or build) Docker images for use in Arvados. (This is currently not possible because running Docker-in-Docker is not allowed unless a container is given special privileges.)
Possible solutions:
h2. Unprivileged build inside a normal container request
There's at least one utility for pulling/manipulating images:
https://github.com/projectatomic/skopeo
However last I checked it doesn't support converting the "docker save" tarfile dump that we use. Support could be added.
For building images, here is a blog post which outlines how to do it with nested unprivileged containers, which is described as an "epic yak shave".
* https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/
This seems like it would require quite a lot of tooling development.
h2. Special container request
Special format container request recognized by crunch-run which executes "docker pull" and/or "docker build" (instead of docker run) and produces the image collection as output. (My preferred solution). Conveniently, now that we have "secrets" support this makes it possible to pull from private repos.
h2. Dedicated docker builder service
New microservice with API for "pull image" and "build image". (Probably inferior to the previous option but mentioned for completeness).
h2. Integrate with external service
For example, docker hub or quay.io. However, pretty sure we don't want to do that. Arvados would not be self-contained, and while it would handle building images it wouldn't actually solve the question of pulling the images into Arvados.
On a related topic, if Arvados supported the Docker registry API (either running Docker registry itself using Keep as a backend or implementing a new service that supports the documented registry API) we could get rid of arv-keepdocker and just use "docker push" and "docker pull".