Project

General

Profile

Actions

Feature #19860

open

Support "pull image" container request

Added by Tom Clegg over 1 year ago. Updated 2 months ago.

Status:
In Progress
Priority:
Normal
Assigned To:
-
Category:
Docker
Target version:
Story points:
3.0

Description

See Build docker images as part of a workflow

A container request like this

container_request:
  container_image: "arvados/builtin" 
  command: ["docker", "pull", "examplerepo:tag"]
  mounts: {}
  runtime_constraints:
    API: true
    RAM: 1000000000
  output_path: "/" 

should pull the indicated image from docker hub and save it to a collection, appropriately tagged, so it can be used in a container request like this

container_request:
  container_image: "examplerepo:tag" 
  # ...

Subtasks 1 (1 open0 closed)

Task #20436: ReviewNewActions

Related issues

Related to Arvados Epics - Idea #16447: Improve container image handlingIn Progress04/01/202406/30/2024Actions
Related to Arvados - Feature #19846: Use collection properties instead of links to tag docker imagesNewTom CleggActions
Related to Arvados - Idea #11724: Pull Docker images without requiring Docker on clientResolvedTom Clegg12/06/2022Actions
Blocks Arvados - Feature #20622: arv-keepdocker uses new "pull image" APINewActions
Actions #1

Updated by Tom Clegg over 1 year ago

  • Related to Idea #16447: Improve container image handling added
Actions #2

Updated by Tom Clegg over 1 year ago

  • Blocked by Feature #19846: Use collection properties instead of links to tag docker images added
Actions #3

Updated by Tom Clegg over 1 year ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Target version set to Future
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Tom Clegg
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Story points set to 3.0
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Target version changed from Future to To be scheduled
Actions #8

Updated by Peter Amstutz about 1 year ago

  • Target version changed from To be scheduled to Development 2023-05-10 sprint
Actions #9

Updated by Tom Clegg about 1 year ago

  • Blocked by deleted (Feature #19846: Use collection properties instead of links to tag docker images)
Actions #10

Updated by Tom Clegg about 1 year ago

  • Related to Feature #19846: Use collection properties instead of links to tag docker images added
Actions #11

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint
Actions #12

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-05-24 sprint to Development 2023-06-07
Actions #13

Updated by Tom Clegg about 1 year ago

  • Related to Idea #11724: Pull Docker images without requiring Docker on client added
Actions #14

Updated by Tom Clegg about 1 year ago

  • Status changed from New to In Progress
Actions #15

Updated by Tom Clegg about 1 year ago

  • Description updated (diff)
Actions #17

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-06-07 to Development 2023-06-21 sprint
Actions #18

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-06-21 sprint to To be scheduled
Actions #19

Updated by Peter Amstutz about 1 year ago

  • Target version changed from To be scheduled to Development 2023-06-21 sprint
Actions #20

Updated by Peter Amstutz about 1 year ago

Make sure to update the "revision" field in the discovery document. (arvados/services/api/app/controllers/arvados/v1/schema_controller.rb)

Actions #21

Updated by Brett Smith about 1 year ago

  • Blocks Feature #20622: arv-keepdocker uses new "pull image" API added
Actions #22

Updated by Brett Smith about 1 year ago

To think about: what's the user experience like when you submit a pull request, then submit a container that uses that image before the pull request has been run?

Actions #23

Updated by Tom Clegg about 1 year ago

  • Assigned To changed from Tom Clegg to Brett Smith

Some notes about the branch in #note-16

Creates an exception to the rule that a container record's container_image must be a PDH: now it can also be "arvados/builtin". I considered using md5("") as the builtin image to preserve the PDH rule, but I think "arvados/builtin" wins by being less mysterious.

If the configured container runtime is singularity instead of docker, you just can't use the "pull image" feature. Another alternative would be to run docker to pull images if the docker executable is available, even when configured to use singularity for running containers. And possibly another config to disable this, because it's annoying to pay/wait for an instance to come up only to find out that pull isn't supported on this cluster.

Actions #24

Updated by Peter Amstutz about 1 year ago

FWIW I believe singularity can pull and convert images natively, so we could have an alternate implementation for singularity. However, we currently treat singularity images as a cached conversion of a docker image and not a source of truth, which makes it impossible to have a completely Dockerless system.

Actions #25

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-06-21 sprint to Development 2023-07-05 sprint
Actions #26

Updated by Peter Amstutz about 1 year ago

  • Assigned To deleted (Brett Smith)
Actions #27

Updated by Peter Amstutz about 1 year ago

  • Target version changed from Development 2023-07-05 sprint to Development 2023-07-19 sprint
Actions #28

Updated by Peter Amstutz 12 months ago

  • Target version changed from Development 2023-07-19 sprint to Development 2023-08-02 sprint
Actions #29

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2023-08-02 sprint to Development 2023-08-16
Actions #30

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2023-08-16 to Development 2023-08-30
Actions #31

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2023-08-30 to To be scheduled
Actions #32

Updated by Peter Amstutz 6 months ago

  • Target version changed from To be scheduled to Development 2024-01-17 sprint
Actions #33

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-01-31 sprint
Actions #34

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-01-31 sprint to Development 2024-02-14 sprint
Actions #35

Updated by Peter Amstutz 6 months ago

  • Target version changed from Development 2024-02-14 sprint to Development 2024-02-28 sprint
Actions #36

Updated by Peter Amstutz 5 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-27 sprint
Actions #37

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #38

Updated by Peter Amstutz 3 months ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-05-08 sprint
Actions #39

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-05-08 sprint to Development 2024-06-05 sprint
Actions #40

Updated by Peter Amstutz 2 months ago

  • Target version changed from Development 2024-06-05 sprint to Future
Actions

Also available in: Atom PDF