Project

General

Profile

Actions

Idea #16447

open

Improve container image handling

Added by Peter Amstutz almost 4 years ago. Updated 3 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Target version:
-
Story points:
-
Release:
Release relationship:
Auto

Description

We want to improve UX for common workflows like:
  • Use Workbench2 (and no shell node) to run a workflow that depends on docker images that are available on dockerhub
  • Build an image from a Dockerfile or git repo, and use that image to run Arvados containers without pushing it to dockerhub
  • Build an image and share it with other users on your cluster without pushing it to dockerhub
  • Share a project that contains a workflow execution plus all of the docker images needed to re-run it even after the referenced images on dockerhub/arvados have been updated or removed
Proposed features:
  • Improve tagging behavior: Avoid inconvenient/mysterious behaviors (e.g., image collection is visible by two users but only one can use it as an image because the tag link isn't shared) by using collection properties instead of tag links to identify docker images. #19846
  • Server-side pull: Workflow runner, when running without access to a local docker daemon (e.g., inside a container submitted by Workbench), can use a container request to pull an externally hosted (dockerhub) image and refer to that image by PDH when executing workflow steps. #11724
  • Server-side build: Workflow runner or other client without access to a local docker daemon (or preferring to leave a better provenance trail) can use a container request to build an image from a specified Dockerfile. #13794
  • Arvados-hosted image namespace: Each cluster has a "docker images" project. Any user with username X can use arv-keepdocker (or a new arvados-client command?) to save docker images in a subproject named X. A container request with container image "arvados:X/Y" will use the image saved in the collection named Y in the subproject named X, subject to usual permissions.
  • Implicit pull at runtime: A container request with container image "docker:abc/def" causes Arvados to fetch/update "abc/def" from dockerhub into an arvados collection, and use that collection PDH as the image in the resulting container.

See Build docker images as part of a workflow


Related issues

Related to Arvados - Idea #11724: Pull Docker images without requiring Docker on clientResolvedTom Clegg12/06/2022Actions
Related to Arvados - Idea #13794: Build Docker images without requiring Docker on clientNewActions
Related to Arvados - Feature #19846: Use collection properties instead of links to tag docker imagesNewTom CleggActions
Related to Arvados - Feature #19860: Support "pull image" container requestIn ProgressActions
Blocks Arvados Epics - Idea #13080: Create/upload workflows through WorkbenchNewActions
Actions #1

Updated by Peter Amstutz almost 4 years ago

  • Start date set to 06/01/2021
  • Due date set to 09/01/2021
Actions #2

Updated by Peter Amstutz about 3 years ago

  • Start date changed from 06/01/2021 to 01/01/2021
  • Due date changed from 09/01/2021 to 06/30/2021
Actions #3

Updated by Peter Amstutz about 3 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Peter Amstutz about 3 years ago

  • Start date changed from 01/01/2021 to 09/01/2021
  • Due date changed from 06/30/2021 to 12/31/2021
Actions #5

Updated by Tom Clegg almost 3 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz over 2 years ago

  • Start date changed from 09/01/2021 to 10/01/2021
  • Due date changed from 12/31/2021 to 01/31/2022
Actions #7

Updated by Peter Amstutz over 2 years ago

  • Start date changed from 10/01/2021 to 01/01/2022
  • Due date changed from 01/31/2022 to 03/31/2022
Actions #8

Updated by Peter Amstutz over 2 years ago

  • Start date changed from 01/01/2022 to 06/01/2022
  • Due date changed from 03/31/2022 to 09/30/2022
Actions #9

Updated by Peter Amstutz about 2 years ago

  • Start date changed from 06/01/2022 to 08/01/2022
  • Due date changed from 09/30/2022 to 11/30/2022
Actions #10

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 08/01/2022 to 09/01/2022
  • Due date changed from 11/30/2022 to 12/31/2022
Actions #11

Updated by Peter Amstutz over 1 year ago

  • Related to Idea #11724: Pull Docker images without requiring Docker on client added
Actions #12

Updated by Peter Amstutz over 1 year ago

  • Related to Idea #13794: Build Docker images without requiring Docker on client added
Actions #13

Updated by Peter Amstutz over 1 year ago

  • Start date changed from 09/01/2022 to 11/01/2022
Actions #14

Updated by Tom Clegg over 1 year ago

  • Related to Feature #19846: Use collection properties instead of links to tag docker images added
Actions #15

Updated by Tom Clegg over 1 year ago

  • Description updated (diff)
Actions #16

Updated by Tom Clegg over 1 year ago

  • Related to Feature #19860: Support "pull image" container request added
Actions #17

Updated by Peter Amstutz about 1 year ago

  • Due date changed from 12/31/2022 to 05/31/2023
Actions #18

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 11/01/2022 to 03/01/2023
  • Due date changed from 05/31/2023 to 07/30/2023
Actions #19

Updated by Peter Amstutz about 1 year ago

  • Start date changed from 03/01/2023 to 05/01/2023
Actions #20

Updated by Peter Amstutz 12 months ago

  • Due date changed from 07/30/2023 to 06/30/2023
Actions #21

Updated by Peter Amstutz 9 months ago

  • Due date changed from 06/30/2023 to 09/30/2023
Actions #22

Updated by Peter Amstutz 6 months ago

  • Due date changed from 09/30/2023 to 02/29/2024
Actions #23

Updated by Peter Amstutz 6 months ago

  • Start date changed from 05/01/2023 to 11/01/2023
Actions #24

Updated by Peter Amstutz 3 months ago

  • Blocks Idea #13080: Create/upload workflows through Workbench added
Actions #25

Updated by Peter Amstutz 3 months ago

  • Due date changed from 02/29/2024 to 03/31/2024
Actions #26

Updated by Peter Amstutz 3 months ago

  • Start date changed from 11/01/2023 to 02/29/2024
  • Due date changed from 03/31/2024 to 04/30/2024
Actions #27

Updated by Peter Amstutz 3 months ago

  • Due date changed from 04/30/2024 to 05/31/2024
Actions #28

Updated by Peter Amstutz 3 months ago

  • Start date changed from 02/29/2024 to 04/01/2024
  • Due date changed from 05/31/2024 to 06/30/2024
Actions

Also available in: Atom PDF