Project

General

Profile

Actions

Idea #16447

open

Improve container image handling

Added by Peter Amstutz almost 4 years ago. Updated 4 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Target version:
-
Story points:
-
Release:
Release relationship:
Auto

Description

We want to improve UX for common workflows like:
  • Use Workbench2 (and no shell node) to run a workflow that depends on docker images that are available on dockerhub
  • Build an image from a Dockerfile or git repo, and use that image to run Arvados containers without pushing it to dockerhub
  • Build an image and share it with other users on your cluster without pushing it to dockerhub
  • Share a project that contains a workflow execution plus all of the docker images needed to re-run it even after the referenced images on dockerhub/arvados have been updated or removed
Proposed features:
  • Improve tagging behavior: Avoid inconvenient/mysterious behaviors (e.g., image collection is visible by two users but only one can use it as an image because the tag link isn't shared) by using collection properties instead of tag links to identify docker images. #19846
  • Server-side pull: Workflow runner, when running without access to a local docker daemon (e.g., inside a container submitted by Workbench), can use a container request to pull an externally hosted (dockerhub) image and refer to that image by PDH when executing workflow steps. #11724
  • Server-side build: Workflow runner or other client without access to a local docker daemon (or preferring to leave a better provenance trail) can use a container request to build an image from a specified Dockerfile. #13794
  • Arvados-hosted image namespace: Each cluster has a "docker images" project. Any user with username X can use arv-keepdocker (or a new arvados-client command?) to save docker images in a subproject named X. A container request with container image "arvados:X/Y" will use the image saved in the collection named Y in the subproject named X, subject to usual permissions.
  • Implicit pull at runtime: A container request with container image "docker:abc/def" causes Arvados to fetch/update "abc/def" from dockerhub into an arvados collection, and use that collection PDH as the image in the resulting container.

See Build docker images as part of a workflow


Related issues

Related to Arvados - Idea #11724: Pull Docker images without requiring Docker on clientResolvedTom Clegg12/06/2022Actions
Related to Arvados - Idea #13794: Build Docker images without requiring Docker on clientNewActions
Related to Arvados - Feature #19846: Use collection properties instead of links to tag docker imagesNewTom CleggActions
Related to Arvados - Feature #19860: Support "pull image" container requestIn ProgressActions
Blocks Arvados Epics - Idea #13080: Create/upload workflows through WorkbenchNewActions
Actions

Also available in: Atom PDF