[Keep] Support multiple storage classes
As an Arvados system administrator, I want to take advantage of the cool/cold storage classes offered by cloud vendors.
This involves designating desired storage class(es) in some way (collection, all collections in a project, etc) as well as a way to migrate between storage classes.
- Each keep volume offers one or more storage classes (the default is just the "default" class).
- Each collection has one or more desired storage classes (the default is just the "default" class).
- When writing, clients may specify one or more required storage classes; if not, the required class is "default". A keepstore server will only write the data on a volume that offers all of the required classes.
- Keep-balance moves data to volumes that have the desired attributes and updates collection records to reflect the storage classes currently satisfied by all blocks in the collection (much like replication level, these are not necessarily equal to the desired classes).
If overlapping collections (i.e., with common data blocks) request different storage classes, keep-balance will maintain multiple copies of the common blocks if necessary to satisfy all collections' requirements.Simplifying restrictions in initial implementation (#12708):
- No client side support.
- No keepstore support for writing data to a given storage class.
- API server configuration specifies the set of classes that can be requested (in addition to "default", which is always available).
- Keepstore configuration specifies the set of classes offered by each volume.
Updated by Tom Clegg about 6 years ago
- specify desired functionality (user can push X buttons, result is Y)
- enumerate features/improvements needed to achieve desired functionality, and define any new APIs needed
- estimate points (or maybe split into separate stories if needed)
Updated by Peter Amstutz almost 6 years ago
Updated by Tom Clegg almost 6 years ago
- Subject changed from [Keep] Support multiple storage tiers to [Keep] Support multiple storage classes
- Description updated (diff)
Pricing examples (moved from description)
Microsoft Azure (pricing at 50-500 TB level)
- LRS-COOL $0.01/GB/mo, $0.01/10Kops + $0.01GB
- LRS-HOT $0.0177/GB/mo, $0.05/10Kops
- GRS-COOL $0.02/GB/mo, $0.20/10Kops + $0.01/GB
- GRS-HOT $0.0354/GB/mo, $0.10/10Kops
- RAGRS-COOL $0.025/GB/mo, $0.20/10Kops + $0.01/GB
- RAGRS-HOST $0.0442/GB/mo, $0.10/10Kops
- Standard - $0.022/GB, $0.004/10Kops (get)
- Infrequent Access - $0.0125/GB, $0.01/10Kops (get)
- Glacier - $0.004/GB + variable retrieval charge depending on speed
- Multi-Regional Storage $0.026/GB/mo
- Regional Storage $0.02/GB/mo
- Nearline Storage $0.01/GB/mo, $0.01/GB retrieval charge
- Coldline Storage $0.007, $0.05/GB retrieval charge
- Optional bucket versioning