Project

General

Profile

Actions

Bug #17390

closed

Set storage classes for intermediates and final outputs

Added by Nico César almost 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Target version:
Start date:
08/12/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

  --storage-classes STORAGE_CLASSES
                        Specify comma separated list of storage classes to be
                        used when saving workflow output to Keep.

base casecase

arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml --storage-classes STORAGE_CLASSES

Existing behavior is final output collection goes to the storage class.

Currently all intermediates go to 'default'.

Expected behaviour:

  • Need to be able to set the storage class of output of a container request
  • Should be able to choose a storage class and have it used for intermediate results through the workflow
  • Need to be able to specify the storage class for a given workflow step ???

In CWL:

StorageClassHint:
  storageClasses: [foo, bar]

Turns this into "output_storage_classes" on the container request.

Command line parameter applies to everything.

Want to support a case where all the intermediates go to fast storage and only the final outputs go into long term storage.


Subtasks 1 (0 open1 closed)

Task #17976: Review 17390-cwl-storage-classesResolvedPeter Amstutz08/12/2021

Actions

Related issues

Related to Arvados Epics - Story #16107: Storage classesResolved03/01/202109/30/2021

Actions
Blocked by Arvados - Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired".Rejected

Actions
Actions #1

Updated by Nico César almost 2 years ago

  • Blocked by Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired". added
Actions #2

Updated by Nico César almost 2 years ago

Actions #3

Updated by Nico César almost 2 years ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz almost 2 years ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Target version set to 2021-08-18 sprint
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Category set to CWL
  • Subject changed from [arvados-cwl-runner] Storage classes revisit to Set storage classes for intermediates and final outputs
Actions #8

Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Peter Amstutz
Actions #9

Updated by Peter Amstutz over 1 year ago

  • Status changed from New to In Progress
Actions #10

Updated by Peter Amstutz over 1 year ago

17390-cwl-storage-classes @ 82c5c20ba8966a23fdfb040aac46a25ff74cfc7b

  • add arv:OutputStorageClass
  • add --intermediate-storage-classes
  • tests check that storage classes propagate to output_storage_classes on container_request

developer-run-tests: #2636

Actions #11

Updated by Lucas Di Pentima over 1 year ago

Actions #12

Updated by Peter Amstutz over 1 year ago

Lucas Di Pentima wrote:

  • I think adding these new hints to the docs is missing.

Will do.

I wrote this response to Tom several days ago but forgot to hit submit:

https://dev.arvados.org/issues/17696#note-11

It seems like it is better to be explicit, since it needs to be aware of storage classes anyway?

Actions #13

Updated by Peter Amstutz over 1 year ago

  • Updated docs

17390-cwl-storage-classes @ 3a4719dd4ffcc1cd92d270d6b698c7ec13678c55

Actions #14

Updated by Lucas Di Pentima over 1 year ago

Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?

Apart from that, it LGTM.

Actions #15

Updated by Peter Amstutz over 1 year ago

  • Status changed from In Progress to Resolved
Actions #16

Updated by Peter Amstutz over 1 year ago

Lucas Di Pentima wrote:

Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?

Yea that was an oversight. Added.

Apart from that, it LGTM.

Thanks, merged.

Actions #17

Updated by Peter Amstutz about 1 year ago

  • Release set to 42
Actions

Also available in: Atom PDF