Project

General

Profile

Actions
Copy link

Feature #19860

open

Support "pull image" container request

Added by Tom Clegg almost 2 years ago. Updated about 19 hours ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
Development 2024-12-18
Story points:
3.0

Description

See Build docker images as part of a workflow

A container request like this

container_request:
  container_image: "arvados/builtin" 
  command: ["docker", "pull", "examplerepo:tag"]
  mounts: {}
  runtime_constraints:
    API: true
    RAM: 1000000000
  output_path: "/"

should pull the indicated image from docker hub and save it to a collection, appropriately tagged, so it can be used in a container request like this

container_request:
  container_image: "examplerepo:tag" 
  # ...

Subtasks 1 (1 open — 0 closed)

Task #20436: ReviewNewActions

Related issues

Related to Arvados Epics - Idea #16447: Improve container image handlingIn Progress12/31/202406/30/2025Actions
Related to Arvados - Feature #19846: Use collection properties instead of links to tag docker imagesNewTom CleggActions
Related to Arvados - Idea #11724: Pull Docker images without requiring Docker on clientResolvedTom Clegg12/06/2022Actions
Blocks Arvados - Feature #20622: arv-keepdocker uses new "pull image" APINewActions
Actions
Copy link
 #1

Updated by Tom Clegg almost 2 years ago

  • Related to Idea #16447: Improve container image handling added
Actions
Copy link
 #2

Updated by Tom Clegg almost 2 years ago

  • Blocked by Feature #19846: Use collection properties instead of links to tag docker images added
Actions
Copy link
 #3

Updated by Tom Clegg almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Peter Amstutz almost 2 years ago

  • Target version set to Future
Actions
Copy link
 #5

Updated by Peter Amstutz almost 2 years ago

  • Assigned To set to Tom Clegg
Actions
Copy link
 #6

Updated by Peter Amstutz over 1 year ago

  • Story points set to 3.0
Actions
Copy link
 #7

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Future to To be scheduled
Actions
Copy link
 #8

Updated by Peter Amstutz over 1 year ago

  • Target version changed from To be scheduled to Development 2023-05-10 sprint
Actions
Copy link
 #9

Updated by Tom Clegg over 1 year ago

  • Blocked by deleted (Feature #19846: Use collection properties instead of links to tag docker images)
Actions
Copy link
 #10

Updated by Tom Clegg over 1 year ago

  • Related to Feature #19846: Use collection properties instead of links to tag docker images added
Actions
Copy link
 #11

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint
Actions
Copy link
 #12

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-05-24 sprint to Development 2023-06-07
Actions
Copy link
 #13

Updated by Tom Clegg over 1 year ago

  • Related to Idea #11724: Pull Docker images without requiring Docker on client added
Actions
Copy link
 #14

Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress
Actions
Copy link
 #15

Updated by Tom Clegg over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #17

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-06-07 to Development 2023-06-21 sprint
Actions
Copy link
 #18

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-06-21 sprint to To be scheduled
Actions
Copy link
 #19

Updated by Peter Amstutz over 1 year ago

  • Target version changed from To be scheduled to Development 2023-06-21 sprint
Actions
Copy link
 #20

Updated by Peter Amstutz over 1 year ago

Make sure to update the "revision" field in the discovery document. (arvados/services/api/app/controllers/arvados/v1/schema_controller.rb)

Actions
Copy link
 #21

Updated by Brett Smith over 1 year ago

  • Blocks Feature #20622: arv-keepdocker uses new "pull image" API added
Actions
Copy link
 #22

Updated by Brett Smith over 1 year ago

To think about: what's the user experience like when you submit a pull request, then submit a container that uses that image before the pull request has been run?

Actions
Copy link
 #23

Updated by Tom Clegg over 1 year ago

  • Assigned To changed from Tom Clegg to Brett Smith

Some notes about the branch in #note-16

Creates an exception to the rule that a container record's container_image must be a PDH: now it can also be "arvados/builtin". I considered using md5("") as the builtin image to preserve the PDH rule, but I think "arvados/builtin" wins by being less mysterious.

If the configured container runtime is singularity instead of docker, you just can't use the "pull image" feature. Another alternative would be to run docker to pull images if the docker executable is available, even when configured to use singularity for running containers. And possibly another config to disable this, because it's annoying to pay/wait for an instance to come up only to find out that pull isn't supported on this cluster.

Actions
Copy link
 #24

Updated by Peter Amstutz over 1 year ago

FWIW I believe singularity can pull and convert images natively, so we could have an alternate implementation for singularity. However, we currently treat singularity images as a cached conversion of a docker image and not a source of truth, which makes it impossible to have a completely Dockerless system.

Actions
Copy link
 #25

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-06-21 sprint to Development 2023-07-05 sprint
Actions
Copy link
 #26

Updated by Peter Amstutz over 1 year ago

  • Assigned To deleted (Brett Smith)
Actions
Copy link
 #27

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-07-05 sprint to Development 2023-07-19 sprint
Actions
Copy link
 #28

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-07-19 sprint to Development 2023-08-02 sprint
Actions
Copy link
 #29

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-08-02 sprint to Development 2023-08-16
Actions
Copy link
 #30

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-08-16 to Development 2023-08-30
Actions
Copy link
 #31

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Development 2023-08-30 to To be scheduled
Actions
Copy link
 #32

Updated by Peter Amstutz 11 months ago

  • Target version changed from To be scheduled to Development 2024-01-17 sprint
Actions
Copy link
 #33

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-01-31 sprint
Actions
Copy link
 #34

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-01-31 sprint to Development 2024-02-14 sprint
Actions
Copy link
 #35

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Actions
Copy link
 #36

Updated by Peter Amstutz 10 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-27 sprint
Actions
Copy link
 #37

Updated by Peter Amstutz 9 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions
Copy link
 #38

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-05-08 sprint
Actions
Copy link
 #39

Updated by Peter Amstutz 8 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-06-05 sprint
Actions
Copy link
 #40

Updated by Peter Amstutz 7 months ago

  • Target version changed from Development 2024-06-05 sprint to Future
Actions
Copy link
 #41

Updated by Peter Amstutz 30 days ago

  • Target version changed from Future to Development 2024-11-20
Actions
Copy link
 #42

Updated by Peter Amstutz 16 days ago

  • Target version changed from Development 2024-11-20 to Development 2024-12-04
Actions
Copy link
 #43

Updated by Peter Amstutz about 19 hours ago

  • Target version changed from Development 2024-12-04 to Development 2024-12-18
Actions
Copy link
 #44

Updated by Peter Amstutz about 19 hours ago

  • Category changed from Docker to Crunch
Actions
Copy link

Also available in: Atom PDF