Bug #17390

Set storage classes for intermediates and final outputs

Added by Nico César 7 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
CWL
Target version:
Start date:
08/12/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

  --storage-classes STORAGE_CLASSES
                        Specify comma separated list of storage classes to be
                        used when saving workflow output to Keep.

base casecase

arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml --storage-classes STORAGE_CLASSES

Existing behavior is final output collection goes to the storage class.

Currently all intermediates go to 'default'.

Expected behaviour:

  • Need to be able to set the storage class of output of a container request
  • Should be able to choose a storage class and have it used for intermediate results through the workflow
  • Need to be able to specify the storage class for a given workflow step ???

In CWL:

StorageClassHint:
  storageClasses: [foo, bar]

Turns this into "output_storage_classes" on the container request.

Command line parameter applies to everything.

Want to support a case where all the intermediates go to fast storage and only the final outputs go into long term storage.


Subtasks

Task #17976: Review 17390-cwl-storage-classesResolvedPeter Amstutz


Related issues

Related to Arvados Epics - Story #16107: Storage classesNew03/01/202109/30/2021

Blocked by Arvados - Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired".Rejected

Associated revisions

Revision 5dc5645a
Added by Peter Amstutz about 1 month ago

Merge branch '17390-cwl-storage-classes' into main

refs #17390

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Nico César 7 months ago

  • Blocked by Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired". added

#2 Updated by Nico César 7 months ago

#3 Updated by Nico César 7 months ago

  • Description updated (diff)

#4 Updated by Peter Amstutz 7 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 2 months ago

  • Target version set to 2021-08-18 sprint

#6 Updated by Peter Amstutz about 2 months ago

  • Description updated (diff)

#7 Updated by Peter Amstutz about 2 months ago

  • Category set to CWL
  • Subject changed from [arvados-cwl-runner] Storage classes revisit to Set storage classes for intermediates and final outputs

#8 Updated by Peter Amstutz about 2 months ago

  • Assigned To set to Peter Amstutz

#9 Updated by Peter Amstutz about 1 month ago

  • Status changed from New to In Progress

#10 Updated by Peter Amstutz about 1 month ago

17390-cwl-storage-classes @ 82c5c20ba8966a23fdfb040aac46a25ff74cfc7b

  • add arv:OutputStorageClass
  • add --intermediate-storage-classes
  • tests check that storage classes propagate to output_storage_classes on container_request

https://ci.arvados.org/view/Developer/job/developer-run-tests/2636/

#11 Updated by Lucas Di Pentima about 1 month ago

#12 Updated by Peter Amstutz about 1 month ago

Lucas Di Pentima wrote:

  • I think adding these new hints to the docs is missing.

Will do.

I wrote this response to Tom several days ago but forgot to hit submit:

https://dev.arvados.org/issues/17696#note-11

It seems like it is better to be explicit, since it needs to be aware of storage classes anyway?

#13 Updated by Peter Amstutz about 1 month ago

  • Updated docs

17390-cwl-storage-classes @ 3a4719dd4ffcc1cd92d270d6b698c7ec13678c55

#14 Updated by Lucas Di Pentima about 1 month ago

Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?

Apart from that, it LGTM.

#15 Updated by Peter Amstutz about 1 month ago

  • Status changed from In Progress to Resolved

#16 Updated by Peter Amstutz about 1 month ago

Lucas Di Pentima wrote:

Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?

Yea that was an oversight. Added.

Apart from that, it LGTM.

Thanks, merged.

Also available in: Atom PDF