Idea #11724
Updated by Peter Amstutz about 6 years ago
Web-only Some users do not have access to Docker on the client (in addition, some shell users don't have access to Docker for security reasons, for example visitors to cloud.curoverse.com). client. Ther It should be a mechanism possible to submit a request to compute node job which can pull a (and/or build) Docker image images for use in Arvados. (This is currently not possible because running Docker-in-Docker is not allowed unless a container is given special privileges.) Possible solutions: h2. Unprivileged pull build inside a normal container request There's at least one utility for pulling/manipulating images: https://github.com/projectatomic/skopeo However last I checked it doesn't support conversion to the "docker save" tarfile dump that we use. Maybe we Support could add support. be added. For building images, here is a blog post which outlines how to do it with nested unprivileged containers, which is described as an "epic yak shave". * https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/ This seems like it would require quite a lot of tooling development. h2. Special container request Special format container request recognized by crunch-run which executes "docker pull" and/or "docker build" (instead of "docker run") docker run) and produces the image collection as output. (My preferred solution). Conveniently, now that we have "secrets" support this makes it possible to pull from private repos. h2. Dedicated docker pull builder service New microservice with API for "pull image" and "build image". This (Probably inferior to the previous option but mentioned for completeness). h2. Integrate with external service For example, docker hub or quay.io. However, pretty sure we don't want to do that. Arvados would avoid not be self-contained, and while it would handle building images it wouldn't actually solve the overhead question of starting up pulling the images into Arvados. On a dedicated VM to run a download process that usually only takes a few seconds. h2. related topic, if Arvados supported the Docker registry API (either running Docker registry itself using Keep as a backend or implementing a new service Deploy https://github.com/docker/distribution or implement that supports the API https://docs.docker.com/registry/spec/api/ . Store layers in keep instead documented registry API) we could get rid of whole image tarballs. Regular arv-keepdocker and just use "docker push" and "docker pull" works. Unprivileged import is more tractable by avoiding format conversion. Additional consideration: to access private registries, we need to provide credentials. Secrets handling is available for container requests. pull".