Feature #17395

Control storage class of container / container_request output

Added by Peter Amstutz about 1 year ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
API
Target version:
Start date:
06/30/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

New field on container called output_storage_classes
New field on container_request called output_storage_classes

When accepting a container request and creating a new container, the new container should take the storage classes from the request. Crunch-run will ensure that container outputs are sent to that storage class.

The output collection of the file will be assigned the storage classes from output_storage_classes

  • crunch-run will respect the value of output_storage_classes on the container record when writing the output collection.

When we create the container request output collection (copied from the container output) the new collection is assigned the storage classes from the container_request output_storage_classes


Subtasks

Task #17797: Review 17395-container-output-storage-classResolvedPeter Amstutz


Related issues

Related to Arvados Epics - Story #16107: Storage classesResolved03/01/202109/30/2021

Associated revisions

Revision 89f541d2
Added by Peter Amstutz 11 months ago

Merge branch '17395-container-output-storage-class' into main

refs #17395

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz about 1 year ago

#2 Updated by Peter Amstutz about 1 year ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 11 months ago

  • Target version set to 2021-06-23 sprint

#5 Updated by Peter Amstutz 11 months ago

  • Assigned To set to Peter Amstutz

#6 Updated by Peter Amstutz 11 months ago

  • Category set to API

#7 Updated by Peter Amstutz 11 months ago

  • Target version changed from 2021-06-23 sprint to 2021-07-07 sprint

#8 Updated by Peter Amstutz 11 months ago

17395-container-output-storage-class @ 4dfa520bd6eb1c594c83f85431d6a39e340fa9cb

  • Migration to add output_storage_classes to container and container_request
  • Add OutputStorageClasses to Container and ContainerRequest in Go SDK
  • Use Container.OutputStorageClasses to set the storage classes used by the "dispatcher" and "container" keep clients used in crunch-run.
  • Add tests & update API documentation.

developer-run-tests: #2560

#9 Updated by Peter Amstutz 11 months ago

  • Status changed from New to In Progress

#10 Updated by Lucas Di Pentima 11 months ago

Some comments below:

  • It seems that the migration file is missing.
  • There's a sentence on the docs: "This feature does not provide a hard guarantee on where data will be stored. Data may be written to default storage and moved to the desired storage class later. If controlling data locality is a hard requirement (such as legal restrictions on the location of data) we recommend setting up multiple Arvados clusters." at the end of https://doc.arvados.org/v2.2/admin/storage-classes.html -- not sure if this story is the right one but I think we'll need to fix it as soon it won't be completely correct.
  • Somewhat related: Should keepproxy use the newly created SetStorageClasses() function at services/keepproxy/keepproxy.go Line 477? Also, some tests could use it.
  • Other than that, it LGTM.

#11 Updated by Peter Amstutz 11 months ago

  • Status changed from In Progress to Resolved

#12 Updated by Peter Amstutz 6 months ago

  • Release set to 42

Also available in: Atom PDF