Bug #11724

Import Docker images without requiring Docker on client

Added by Peter Amstutz 11 months ago. Updated 6 days ago.

Assigned To:
Target version:
Start date:
Due date:
% Done:


Estimated time:
Story points:


Some users do not have access to Docker on the client. It should be possible to submit a compute node job which can pull (and/or build) Docker images for use in Arvados. (This is currently not possible because running Docker-in-Docker is not allowed unless a container is given special privileges.)

Possible solutions:

Unprivileged build inside a normal container request

There's at least one utility for pulling/manipulating images:


However last I checked it doesn't support the "docker save" tarfile dump that we use. Support could be added.

For building images, here is a blog post which outlines how to do it with nested unprivileged containers, which is described as an "epic yak shave".

This seems like it would require quite a lot of tooling development.

Special container request

Special format container request recognized by crunch-run which executes "docker pull" and/or "docker build" (instead of docker run) and produces the image collection as output. (My preferred solution). Conveniently, now that we have "secrets" support this makes it possible to pull from private repos.

Dedicated docker builder service

New microservice with API for "pull image" and "build image". (Probably inferior to the previous option but mentioned for completeness).

Integrate with external service

For example, docker hub or quay.io. However, pretty sure we don't want to do that. Arvados would not be self-contained, and while it would handle building images it wouldn't actually solve the question of pulling the images into Arvados.

On a related topic, if Arvados supported the Docker registry API (either running Docker registry itself using Keep as a backend or implementing a new service that supports the documented registry API) we could get rid of arv-keepdocker and just use "docker push" and "docker pull".

Related issues

Related to Arvados - Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.comNew


#1 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#2 Updated by Tom Morris 8 months ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Clegg 7 days ago

  • Related to Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.com added

#4 Updated by Peter Amstutz 6 days ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 6 days ago

  • Description updated (diff)

Also available in: Atom PDF