Bug #11724

Import Docker images without requiring Docker on client

Added by Peter Amstutz about 1 year ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Some users do not have access to Docker on the client. It should be possible to submit a compute node job which can pull (and/or build) Docker images for use in Arvados. (This is currently not possible because running Docker-in-Docker is not allowed unless a container is given special privileges.)

Possible solutions:

Unprivileged build inside a normal container request

There's at least one utility for pulling/manipulating images:

https://github.com/projectatomic/skopeo

However last I checked it doesn't support the "docker save" tarfile dump that we use. Support could be added.

For building images, here is a blog post which outlines how to do it with nested unprivileged containers, which is described as an "epic yak shave".

This seems like it would require quite a lot of tooling development.

Special container request

Special format container request recognized by crunch-run which executes "docker pull" and/or "docker build" (instead of docker run) and produces the image collection as output. (My preferred solution). Conveniently, now that we have "secrets" support this makes it possible to pull from private repos.

Dedicated docker builder service

New microservice with API for "pull image" and "build image". (Probably inferior to the previous option but mentioned for completeness).

Integrate with external service

For example, docker hub or quay.io. However, pretty sure we don't want to do that. Arvados would not be self-contained, and while it would handle building images it wouldn't actually solve the question of pulling the images into Arvados.

On a related topic, if Arvados supported the Docker registry API (either running Docker registry itself using Keep as a backend or implementing a new service that supports the documented registry API) we could get rid of arv-keepdocker and just use "docker push" and "docker pull".


Related issues

Related to Arvados - Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.comNew

Is duplicate of Arvados - Story #9046: [Crunch2] Can issue container request to pull and import Docker imagesNew

History

#1 Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)

#2 Updated by Tom Morris 10 months ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Clegg 2 months ago

  • Related to Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.com added

#4 Updated by Peter Amstutz 2 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 2 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz about 1 month ago

  • Is duplicate of Story #9046: [Crunch2] Can issue container request to pull and import Docker images added

Also available in: Atom PDF