Story #16107

Storage classes

Added by Ward Vandewege over 1 year ago. Updated 26 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
03/01/2021
Due date:
09/30/2021
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

Requirements and high level design

Each collection has one or more desired storage classes. The Keep blocks making up the collection inherit the storage classes from the collection.
  • Each keep volume is assigned one or more storage classes
  • The system asynchronously copies/moves blocks between Keep volumes to fulfill the desired storage classes for each block.

When uploading a block, the Python and Go SDKs support passing a list of storage classes to influence placement of the data block.
Each data block will be stored on volumes that fulfill the desired storage classes.

  • A block that has two or more storage classes may be fulfilled by a single volume that fulfills all storage classes, or multiple volumes for each storage class.
  • [#17349] The “replicas_desired” field expresses the number of replicas per storage class.
    • A single volume can be configured to count as multiple replicas (existing behavior)
    • [#17350 for keep-balance] A block assigned to two storage classes with N=2 replicas could have 2, 3 or 4 actual copies. For example, a block requesting storage classes A and B could be written to two volumes with storage classes “A and B”, or two “A” volumes and two “B” volumes, or “A”, “B” and “A,B”.
    • If sufficient replicas cannot be written for each storage class during upload, it is a fatal error (existing behavior)
  • Storage classes are advisory and do not guarantee that a block will only be stored on a certain class or will never be stored on a certain class. Some circumstances where blocks may not be stored on only the requested storage class:
    • User has specified an impossible situation, such as changing a collection to a storage class that isn’t fulfilled by any volume
    • The same block may be referenced by multiple collections with different storage classes
    • A storage class for a collection may have been changed but the system has not caught up to it yet
    • In these cases the block will remain on its original storage volume
    • There will be a reporting tool (keep-balance or other) that reports when there is a mismatch between the desired and actual storage classes for a block. The tool will also provide a way to report which blocks (associated collections) on a certain storage class also exist on a different storage class.
  • When writing a block, the desired storage classes are passed in the keep service request
  • [#17389] Keepproxy will understand storage classes and forwards blocks to the appropriate keepstore service.
  • When reading a collection, the Python and Go SDK support passing a list of storage classes to inform block lookup.
  • There is always a “default” storage class. It is an error if there is not at least one volume with the “default” class.
    • If an upload doesn’t specify a storage class it will use the ‘default’ storage class
  • Container and container request records gain a field specifying the storage class of the output collection.
  • [#17351][#17390] The following Arvados component swill gain support for specifying storage classes for data upload: arv-put, arvados-cwl-runner, crunch-run
    • When writing a block, the desired storage classes are passed in the keep service request
    • Keepproxy will understand storage classes and forwards blocks to the appropriate keepstore service.
    • When reading a collection, the Python and Go SDK support passing a list of storage classes to inform block lookup.
    • There is always a “default” storage class. It is an error if there is not at least one volume with the “default” class.
    • If an upload doesn’t specify a storage class it will use the ‘default’ storage class
    • Container and container request records gain a field specifying the storage class of the output collection.
  • Workbench 2 collection view will display the storage class of the collection.
  • Work will be tested with unit, functional and integration testing using Amazon S3

Related issues

Related to Arvados - Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired".Rejected

Related to Arvados - Feature #17350: [keep-balance] Expected behaviour with different "replicas_desired" and "storage_classes_desired" valuesRejected

Related to Arvados - Feature #17351: [arv-put] Storage classesResolved06/03/2021

Related to Arvados - Feature #17388: [arv-copy] Storage classes revisitResolved07/23/2021

Related to Arvados - Feature #17389: Storage classes support in keepproxyResolved06/18/2021

Related to Arvados - Bug #17390: Set storage classes for intermediates and final outputsResolved08/12/2021

Related to Arvados - Feature #17391: [keepstore] expose the volume storage classes Closed

Related to Arvados - Feature #17392: Support writing blocks to correct storage classes in Go SDKResolved04/12/2021

Related to Arvados - Feature #13382: [keepstore] Write new blocks to appropriate storage classResolved04/02/2021

Related to Arvados - Feature #17394: Go SDK CollectionFS writes files to correct storage classResolved07/15/2021

Related to Arvados - Feature #17395: Control storage class of container / container_request outputResolved06/30/2021

Related to Arvados - Feature #17393: Go and Python SDK propagate correct storage class to keepstoreDuplicate

Related to Arvados - Support #17447: Scoping/grooming storage classesResolved

Related to Arvados - Story #17465: Support writing blocks to correct storage classes in Python SDKResolved06/01/2021

Related to Arvados - Feature #11184: [Keep] Support multiple storage classesResolved

Related to Arvados - Feature #17572: arv-mount understands storage classesResolved06/21/2021

Related to Arvados - Story #17573: User interface for exposing / changing storage classes on a collectionResolved07/15/2021

Related to Arvados - Feature #17574: keep-balance updates API server with correct storage_classes_confirmedResolved07/29/2021

Related to Arvados - Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overriddenResolved08/19/2021

Related to Arvados - Story #17697: Design for reporting tools to determine what data is on multiple storage classes.Resolved

Related to Arvados - Feature #17698: Parallelize writes inside keepstore when there are writes that request multiple storage classes.Resolved08/13/2021

Related to Arvados - Feature #17967: Prioritize reads from different storage classesResolved08/05/2021

Blocked by Arvados - Feature #17993: [deduplication-report] supports storage classesNew

Blocked by Arvados - Feature #17994: [api] storage class fields should be supported in filtersResolved08/27/2021

Blocked by Arvados - Feature #17995: [api] add method to get collections where replication_confirmed < replication_desiredResolved08/27/2021

History

#1 Updated by Peter Amstutz over 1 year ago

  • Start date changed from 08/01/2020 to 10/01/2020
  • Due date changed from 11/30/2020 to 12/31/2020

#2 Updated by Peter Amstutz over 1 year ago

  • Start date changed from 10/01/2020 to 11/01/2020
  • Due date changed from 12/31/2020 to 01/31/2021

#3 Updated by Peter Amstutz about 1 year ago

  • Start date changed from 11/01/2020 to 02/01/2021
  • Due date changed from 01/31/2021 to 04/30/2021

#4 Updated by Peter Amstutz 12 months ago

  • Subject changed from Storage classes to Storage classes / glacier

#5 Updated by Peter Amstutz 12 months ago

  • Start date changed from 02/01/2021 to 01/01/2021
  • Due date changed from 04/30/2021 to 03/31/2021

#6 Updated by Peter Amstutz 10 months ago

  • Due date changed from 03/31/2021 to 06/30/2021

#7 Updated by Peter Amstutz 9 months ago

  • Subject changed from Storage classes / glacier to Storage classes

#8 Updated by Peter Amstutz 9 months ago

  • Start date changed from 01/01/2021 to 03/01/2021

#9 Updated by Nico César 7 months ago

  • Description updated (diff)

#10 Updated by Nico César 7 months ago

  • Related to Feature #17349: [Keep API] Revisit "replicas_desired" and "storage_classes_desired". added

#11 Updated by Nico César 7 months ago

  • Related to Feature #17350: [keep-balance] Expected behaviour with different "replicas_desired" and "storage_classes_desired" values added

#12 Updated by Nico César 7 months ago

  • Description updated (diff)

#13 Updated by Nico César 7 months ago

#14 Updated by Nico César 7 months ago

#15 Updated by Nico César 7 months ago

  • Related to Feature #17389: Storage classes support in keepproxy added

#16 Updated by Nico César 7 months ago

  • Description updated (diff)

#17 Updated by Nico César 7 months ago

  • Related to Bug #17390: Set storage classes for intermediates and final outputs added

#18 Updated by Nico César 7 months ago

  • Description updated (diff)

#19 Updated by Nico César 7 months ago

  • Description updated (diff)

#20 Updated by Nico César 7 months ago

  • Related to Feature #17391: [keepstore] expose the volume storage classes added

#21 Updated by Nico César 7 months ago

  • Related to Feature #17392: Support writing blocks to correct storage classes in Go SDK added

#22 Updated by Nico César 7 months ago

  • Related to Feature #13382: [keepstore] Write new blocks to appropriate storage class added

#23 Updated by Peter Amstutz 7 months ago

  • Related to Feature #17394: Go SDK CollectionFS writes files to correct storage class added

#24 Updated by Peter Amstutz 7 months ago

  • Related to Feature #17395: Control storage class of container / container_request output added

#25 Updated by Peter Amstutz 7 months ago

  • Related to Feature #17393: Go and Python SDK propagate correct storage class to keepstore added

#26 Updated by Peter Amstutz 7 months ago

#27 Updated by Peter Amstutz 6 months ago

  • Related to Story #17465: Support writing blocks to correct storage classes in Python SDK added

#28 Updated by Tom Clegg 5 months ago

  • Related to Feature #11184: [Keep] Support multiple storage classes added

#29 Updated by Peter Amstutz 5 months ago

  • Related to Feature #17572: arv-mount understands storage classes added

#30 Updated by Peter Amstutz 5 months ago

  • Related to Story #17573: User interface for exposing / changing storage classes on a collection added

#31 Updated by Peter Amstutz 5 months ago

  • Related to Feature #17574: keep-balance updates API server with correct storage_classes_confirmed added

#32 Updated by Peter Amstutz 4 months ago

Customer request: can specify desired default storage classes, which can be more that one.

#33 Updated by Peter Amstutz 4 months ago

Reporting tools to determine what data is on multiple storage classes.

#34 Updated by Peter Amstutz 4 months ago

Way to prioritize reading data from volumes with faster storage classes.

#35 Updated by Peter Amstutz 4 months ago

Parallelize writes inside keepstore when there are writes that request multiple storage classes.

#36 Updated by Peter Amstutz 4 months ago

Has use case crunch-run starting its own keepstore server.

#37 Updated by Peter Amstutz 4 months ago

  • Related to Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overridden added

#38 Updated by Peter Amstutz 4 months ago

  • Related to Story #17697: Design for reporting tools to determine what data is on multiple storage classes. added

#39 Updated by Peter Amstutz 4 months ago

  • Related to Feature #17698: Parallelize writes inside keepstore when there are writes that request multiple storage classes. added

#40 Updated by Peter Amstutz 3 months ago

  • Due date changed from 06/30/2021 to 07/31/2021

#41 Updated by Peter Amstutz 3 months ago

  • Due date changed from 07/31/2021 to 08/31/2021

#42 Updated by Peter Amstutz about 2 months ago

  • Related to Feature #17967: Prioritize reads from different storage classes added

#43 Updated by Ward Vandewege about 1 month ago

  • Blocked by Feature #17993: [deduplication-report] supports storage classes added

#44 Updated by Ward Vandewege about 1 month ago

  • Blocked by Feature #17994: [api] storage class fields should be supported in filters added

#45 Updated by Ward Vandewege about 1 month ago

  • Blocked by Feature #17995: [api] add method to get collections where replication_confirmed < replication_desired added

#46 Updated by Peter Amstutz 26 days ago

  • Due date changed from 08/31/2021 to 09/30/2021

Also available in: Atom PDF