Project

General

Profile

Actions

Idea #11724

closed

Pull Docker images without requiring Docker on client

Added by Peter Amstutz almost 7 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Story points:
-

Description

Web-only users do not have access to Docker on the client (in addition, some shell users don't have access to Docker for security reasons, for example visitors to cloud.curoverse.com). Ther should be a mechanism to submit a request to pull a Docker image for use in Arvados.

Possible solutions:

Unprivileged pull inside a normal container request

There's at least one utility for pulling/manipulating images:

https://github.com/projectatomic/skopeo

However last I checked it doesn't support conversion to the "docker save" tarfile dump that we use. Maybe we could add support.

Special container request

Special format container request recognized by crunch-run which executes "docker pull" (instead of "docker run") and produces the image collection as output.

Dedicated "docker pull" service and/or WES

New microservice with API for "pull image". This would avoid the overhead of starting up a dedicated VM to run a download process that usually only takes a few seconds.

Note: the Workflow Execution Service (WES) server sort of already does this; if workflow run is submitted that requires pulling Docker images, it will pull them and upload them. This is existing arvados-cwl-runner behavior which normally requires Docker on the client, but in the case of WES, the WES gateway is the agent that runs arvados-cwl-runner and not the original client.

So there's also an option to migrate clients that submit workflows (a-c-r, workbench, composer) to use Arvados WES instead of directly creating container requests.

Arvados Docker registry service

Deploy https://github.com/docker/distribution or implement the API https://docs.docker.com/registry/spec/api/ . Store layers in keep instead of whole image tarballs. Regular "docker push" and "docker pull" works. Unprivileged import is more tractable by avoiding format conversion.

Additional consideration: to access private registries, we need to provide credentials. Secrets handling is available for container requests.


Subtasks 1 (0 open1 closed)

Task #19807: Design proposalResolvedTom Clegg12/06/2022Actions

Related issues

Related to Arvados - Idea #13325: As a CWL learner, I would like to be able to run CWL workflows easily on playgroundNewActions
Related to Arvados Epics - Idea #16447: Improve container image handlingIn Progress04/01/202406/30/2024Actions
Related to Arvados - Idea #13794: Build Docker images without requiring Docker on clientNewActions
Related to Arvados - Feature #19860: Support "pull image" container requestIn ProgressActions
Is duplicate of Arvados - Idea #9046: [Crunch2] Can issue container request to pull and import Docker imagesClosedActions
Blocked by Arvados - Feature #19846: Use collection properties instead of links to tag docker imagesNewTom CleggActions
Actions

Also available in: Atom PDF