Bug #17390
closed
Set storage classes for intermediates and final outputs
Description
--storage-classes STORAGE_CLASSES
Specify comma separated list of storage classes to be
used when saving workflow output to Keep.
base casecase¶
arvados-cwl-runner --name "CWL diagnostics hasher" --create-workflow hasher-workflow.cwl hasher-workflow-job.yml --storage-classes STORAGE_CLASSES
Existing behavior is final output collection goes to the storage class.
Currently all intermediates go to 'default'.
Expected behaviour:
- Need to be able to set the storage class of output of a container request
- Should be able to choose a storage class and have it used for intermediate results through the workflow
- Need to be able to specify the storage class for a given workflow step ???
In CWL:
StorageClassHint: storageClasses: [foo, bar]
Turns this into "output_storage_classes" on the container request.
Command line parameter applies to everything.
Want to support a case where all the intermediates go to fast storage and only the final outputs go into long term storage.
Updated by Nico César almost 4 years ago
- Blocked by Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired". added
Updated by Nico César almost 4 years ago
- Related to Idea #16107: Storage classes added
Updated by Peter Amstutz over 3 years ago
- Target version set to 2021-08-18 sprint
Updated by Peter Amstutz over 3 years ago
- Category set to CWL
- Subject changed from [arvados-cwl-runner] Storage classes revisit to Set storage classes for intermediates and final outputs
Updated by Peter Amstutz over 3 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz over 3 years ago
17390-cwl-storage-classes @ 82c5c20ba8966a23fdfb040aac46a25ff74cfc7b
- add arv:OutputStorageClass
- add --intermediate-storage-classes
- tests check that storage classes propagate to output_storage_classes on container_request
Updated by Lucas Di Pentima over 3 years ago
- I think adding these new hints to the docs is missing.
- Regarding the default classes, it probably doesn't need to check for them: See Tom's comment at https://dev.arvados.org/issues/17696#note-9
Updated by Peter Amstutz over 3 years ago
Lucas Di Pentima wrote:
- I think adding these new hints to the docs is missing.
Will do.
- Regarding the default classes, it probably doesn't need to check for them: See Tom's comment at https://dev.arvados.org/issues/17696#note-9
I wrote this response to Tom several days ago but forgot to hit submit:
https://dev.arvados.org/issues/17696#note-11
It seems like it is better to be explicit, since it needs to be aware of storage classes anyway?
Updated by Peter Amstutz over 3 years ago
- Updated docs
17390-cwl-storage-classes @ 3a4719dd4ffcc1cd92d270d6b698c7ec13678c55
Updated by Lucas Di Pentima over 3 years ago
Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?
Apart from that, it LGTM.
Updated by Peter Amstutz over 3 years ago
- Status changed from In Progress to Resolved
Updated by Peter Amstutz over 3 years ago
Lucas Di Pentima wrote:
Sorry, with the docs comment I meant to paste this link but forgot: https://doc.arvados.org/main/user/cwl/cwl-extensions.html -- Should we add the new hints on our list of CWL extensions? or is it only meant for internal use?
Yea that was an oversight. Added.
Apart from that, it LGTM.
Thanks, merged.