Feature #19860: Support "pull image" container request - Arvados

Actions

Copy link

Feature #19860

open

Support "pull image" container request

Added by Tom Clegg about 1 year ago. Updated 4 months ago.

Status:

In Progress

Priority:

Normal

Assigned To:

Category:

Docker

Target version:

To be scheduled

Start date:

Due date:

% Done:

Estimated time:

(Total: 0.00 h)

Story points:

3.0

Description

See Build docker images as part of a workflow

A container request like this

container_request:
  container_image: "arvados/builtin" 
  command: ["docker", "pull", "examplerepo:tag"]
  mounts: {}
  runtime_constraints:
    API: true
    RAM: 1000000000
  output_path: "/"

should pull the indicated image from docker hub and save it to a collection, appropriately tagged, so it can be used in a container request like this

container_request:
  container_image: "examplerepo:tag" 
  # ...

Subtasks 1 (1 open — 0 closed)

Related issues

Actions

Copy link

Updated by Tom Clegg about 1 year ago

Related to Story #16447: Improve container image handling added

Actions

Copy link

Updated by Tom Clegg about 1 year ago

Blocked by Story #19846: Use collection properties instead of links to tag docker images added

Actions

Copy link

Updated by Tom Clegg 12 months ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Target version set to To be groomed

Actions

Copy link

Updated by Peter Amstutz 11 months ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Peter Amstutz 9 months ago

Story points set to 3.0

Actions

Copy link

Updated by Peter Amstutz 9 months ago

Target version changed from To be groomed to To be scheduled

Actions

Copy link

Updated by Peter Amstutz 8 months ago

Target version changed from To be scheduled to Development 2023-05-10 sprint

Actions

Copy link

Updated by Tom Clegg 8 months ago

Blocked by deleted (Story #19846: Use collection properties instead of links to tag docker images)

Actions

Copy link

#10

Updated by Tom Clegg 8 months ago

Related to Story #19846: Use collection properties instead of links to tag docker images added

Actions

Copy link

#11

Updated by Peter Amstutz 7 months ago

Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint

Actions

Copy link

#12

Updated by Peter Amstutz 7 months ago

Target version changed from Development 2023-05-24 sprint to Development 2023-06-07

Actions

Copy link

#13

Updated by Tom Clegg 7 months ago

Related to Story #11724: Pull Docker images without requiring Docker on client added

Actions

Copy link

#14

Updated by Tom Clegg 7 months ago

Status changed from New to In Progress

Actions

Copy link

#15

Updated by Tom Clegg 7 months ago

Description updated (diff)

Actions

Copy link

#16

Updated by Tom Clegg 7 months ago

19860-docker-pull-container @ 42b266e3667f08b219173c6eb148e1a5500046c1 -- developer-run-tests: #3656

Actions

Copy link

#17

Updated by Peter Amstutz 7 months ago

Target version changed from Development 2023-06-07 to Development 2023-06-21 sprint

Actions

Copy link

#18

Updated by Peter Amstutz 6 months ago

Target version changed from Development 2023-06-21 sprint to To be scheduled

Actions

Copy link

#19

Updated by Peter Amstutz 6 months ago

Target version changed from To be scheduled to Development 2023-06-21 sprint

Actions

Copy link

#20

Updated by Peter Amstutz 6 months ago

Make sure to update the "revision" field in the discovery document. (arvados/services/api/app/controllers/arvados/v1/schema_controller.rb)

Actions

Copy link

#21

Updated by Brett Smith 6 months ago

Blocks Feature #20622: arv-keepdocker uses new "pull image" API added

Actions

Copy link

#22

Updated by Brett Smith 6 months ago

To think about: what's the user experience like when you submit a pull request, then submit a container that uses that image before the pull request has been run?

Actions

Copy link

#23

Updated by Tom Clegg 6 months ago

Assigned To changed from Tom Clegg to Brett Smith

Some notes about the branch in #note-16

Creates an exception to the rule that a container record's container_image must be a PDH: now it can also be "arvados/builtin". I considered using md5("") as the builtin image to preserve the PDH rule, but I think "arvados/builtin" wins by being less mysterious.

If the configured container runtime is singularity instead of docker, you just can't use the "pull image" feature. Another alternative would be to run docker to pull images if the docker executable is available, even when configured to use singularity for running containers. And possibly another config to disable this, because it's annoying to pay/wait for an instance to come up only to find out that pull isn't supported on this cluster.

Actions

Copy link

#24

Updated by Peter Amstutz 6 months ago

FWIW I believe singularity can pull and convert images natively, so we could have an alternate implementation for singularity. However, we currently treat singularity images as a cached conversion of a docker image and not a source of truth, which makes it impossible to have a completely Dockerless system.

Actions

Copy link

#25