Story #11724

Pull Docker images without requiring Docker on client

Added by Peter Amstutz over 1 year ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Web-only users do not have access to Docker on the client (in addition, some shell users don't have access to Docker for security reasons, for example visitors to cloud.curoverse.com). Ther should be a mechanism to submit a request to pull a Docker image for use in Arvados.

Possible solutions:

Unprivileged pull inside a normal container request

There's at least one utility for pulling/manipulating images:

https://github.com/projectatomic/skopeo

However last I checked it doesn't support conversion to the "docker save" tarfile dump that we use. Maybe we could add support.

Special container request

Special format container request recognized by crunch-run which executes "docker pull" (instead of "docker run") and produces the image collection as output.

Dedicated "docker pull" service and/or WES

New microservice with API for "pull image". This would avoid the overhead of starting up a dedicated VM to run a download process that usually only takes a few seconds.

Note: the Workflow Execution Service (WES) server sort of already does this; if workflow run is submitted that requires pulling Docker images, it will pull them and upload them. This is existing arvados-cwl-runner behavior which normally requires Docker on the client, but in the case of WES, the WES gateway is the agent that runs arvados-cwl-runner and not the original client.

So there's also an option to migrate clients that submit workflows (a-c-r, workbench, composer) to use Arvados WES instead of directly creating container requests.

Arvados Docker registry service

Deploy https://github.com/docker/distribution or implement the API https://docs.docker.com/registry/spec/api/ . Store layers in keep instead of whole image tarballs. Regular "docker push" and "docker pull" works. Unprivileged import is more tractable by avoiding format conversion.

Additional consideration: to access private registries, we need to provide credentials. Secrets handling is available for container requests.


Related issues

Related to Arvados - Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.comNew

Related to Arvados - Story #13794: Build Docker images without requiring Docker on clientNew

Is duplicate of Arvados - Story #9046: [Crunch2] Can issue container request to pull and import Docker imagesNew

History

#1 Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)

#2 Updated by Tom Morris over 1 year ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Clegg 8 months ago

  • Related to Story #13325: As a CWL learner, I would like to be able to run CWL workflows easily on cloud.curoverse.com added

#4 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 7 months ago

  • Is duplicate of Story #9046: [Crunch2] Can issue container request to pull and import Docker images added

#7 Updated by Tom Morris 5 months ago

  • Tracker changed from Bug to Story
  • Target version changed from Arvados Future Sprints to To Be Groomed

#8 Updated by Peter Amstutz 5 months ago

  • Subject changed from Import Docker images without requiring Docker on client to Pull Docker images without requiring Docker on client
  • Description updated (diff)

#9 Updated by Peter Amstutz 5 months ago

  • Related to Story #13794: Build Docker images without requiring Docker on client added

#10 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

#11 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

#12 Updated by Peter Amstutz 5 months ago

  • Description updated (diff)

Also available in: Atom PDF