Feature #19860: Support "pull image" container request - Arvados

Custom queries

All assigned issues
All issues assigned for grooming
My issues for grooming
My issues for grooming (no story pts)
Prioritized open issues

Actions

Copy link

Feature #19860

open

Support "pull image" container request

Added by Tom Clegg over 1 year ago. Updated 22 days ago.

Status:

In Progress

Priority:

Normal

Assigned To:

Category:

Docker

Target version:

Future

Story points:

3.0

Description

See Build docker images as part of a workflow

A container request like this

container_request:
  container_image: "arvados/builtin" 
  command: ["docker", "pull", "examplerepo:tag"]
  mounts: {}
  runtime_constraints:
    API: true
    RAM: 1000000000
  output_path: "/"

should pull the indicated image from docker hub and save it to a collection, appropriately tagged, so it can be used in a container request like this

container_request:
  container_image: "examplerepo:tag" 
  # ...

Subtasks 1 (1 open — 0 closed)

Task #20436: Review

New

Actions

Related issues

Related to Arvados Epics - Idea #16447: Improve container image handling	In Progress		04/01/2024	06/30/2024	Actions
Related to Arvados - Feature #19846: Use collection properties instead of links to tag docker images	New	Tom Clegg			Actions
Related to Arvados - Idea #11724: Pull Docker images without requiring Docker on client	Resolved	Tom Clegg	12/06/2022		Actions
Blocks Arvados - Feature #20622: arv-keepdocker uses new "pull image" API	New				Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Tom Clegg over 1 year ago

Related to Idea #16447: Improve container image handling added

Actions

Copy link

Updated by Tom Clegg over 1 year ago

Blocked by Feature #19846: Use collection properties instead of links to tag docker images added

Actions

Copy link

Updated by Tom Clegg over 1 year ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz over 1 year ago

Target version set to Future

Actions

Copy link

Updated by Peter Amstutz over 1 year ago

Assigned To set to Tom Clegg

Actions

Copy link

Updated by Peter Amstutz about 1 year ago

Story points set to 3.0

Actions

Copy link

Updated by Peter Amstutz about 1 year ago

Target version changed from Future to To be scheduled

Actions

Copy link

Updated by Peter Amstutz about 1 year ago

Target version changed from To be scheduled to Development 2023-05-10 sprint

Actions

Copy link

Updated by Tom Clegg about 1 year ago

Blocked by deleted (Feature #19846: Use collection properties instead of links to tag docker images)

Actions

Copy link

#10

Updated by Tom Clegg about 1 year ago

Related to Feature #19846: Use collection properties instead of links to tag docker images added

Actions

Copy link

#11

Updated by Peter Amstutz about 1 year ago

Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint

Actions

Copy link

#12

Updated by Peter Amstutz 12 months ago

Target version changed from Development 2023-05-24 sprint to Development 2023-06-07

Actions

Copy link

#13

Updated by Tom Clegg 12 months ago

Related to Idea #11724: Pull Docker images without requiring Docker on client added

Actions

Copy link

#14

Updated by Tom Clegg 12 months ago

Status changed from New to In Progress

Actions

Copy link

#15

Updated by Tom Clegg 12 months ago

Description updated (diff)

Actions

Copy link

#16

Updated by Tom Clegg 12 months ago

19860-docker-pull-container @ 42b266e3667f08b219173c6eb148e1a5500046c1 -- developer-run-tests: #3656

Actions

Copy link

#17

Updated by Peter Amstutz 11 months ago

Target version changed from Development 2023-06-07 to Development 2023-06-21 sprint

Actions

Copy link

#18

Updated by Peter Amstutz 11 months ago

Target version changed from Development 2023-06-21 sprint to To be scheduled

Actions

Copy link

#19

Updated by Peter Amstutz 11 months ago

Target version changed from To be scheduled to Development 2023-06-21 sprint

Actions

Copy link

#20

Updated by Peter Amstutz 11 months ago

Make sure to update the "revision" field in the discovery document. (arvados/services/api/app/controllers/arvados/v1/schema_controller.rb)

Actions

Copy link

#21

Updated by Brett Smith 11 months ago

Blocks Feature #20622: arv-keepdocker uses new "pull image" API added

Actions

Copy link

#22

Updated by Brett Smith 11 months ago

To think about: what's the user experience like when you submit a pull request, then submit a container that uses that image before the pull request has been run?

Actions

Copy link

#23

Updated by Tom Clegg 11 months ago

Assigned To changed from Tom Clegg to Brett Smith

Some notes about the branch in #note-16

Creates an exception to the rule that a container record's container_image must be a PDH: now it can also be "arvados/builtin". I considered using md5("") as the builtin image to preserve the PDH rule, but I think "arvados/builtin" wins by being less mysterious.

If the configured container runtime is singularity instead of docker, you just can't use the "pull image" feature. Another alternative would be to run docker to pull images if the docker executable is available, even when configured to use singularity for running containers. And possibly another config to disable this, because it's annoying to pay/wait for an instance to come up only to find out that pull isn't supported on this cluster.

Actions

Copy link

#24

Updated by Peter Amstutz 11 months ago

FWIW I believe singularity can pull and convert images natively, so we could have an alternate implementation for singularity. However, we currently treat singularity images as a cached conversion of a docker image and not a source of truth, which makes it impossible to have a completely Dockerless system.

Actions

Copy link

#25