[Keep] Support multiple storage tiers
|Assignee:||Tom Morris||% Done:|
|Target version:||Arvados Future Sprints|
|Story points||3.0||Remaining (hours)||0.00 hour|
|Velocity based estimate||0 days|
As an Arvados system administrator, I'd want the ability take advantage of the cool/cold storage classes offered by cloud vendors.
This would potentially require the ability to designate storage tier in some way (collection, all collections in a project, etc) as well as a way to migrate between storage classes/locations,
Keep volumes have storage attributes that they offer.
Collections have storage attributes that they desire. Blocks get written to same Keep store server as today.
Keep balance moves data to volumes that have the desired attributes and updates the collection to reflect the actual storage class (ie there are separate requested storage class and current storage class). Keep balance needs to track blocks by volume rather than by server as it currently does.
If multiple collections have the same block with different storage classes, the block gets stored multiple times, once in each.
For the initial iteration, there is no client side support.
Microsoft Azure (pricing at 50-500 TB level)
- LRS-COOL $0.01/GB/mo, $0.01/10Kops + $0.01GB
- LRS-HOT $0.0177/GB/mo, $0.05/10Kops
- GRS-COOL $0.02/GB/mo, $0.20/10Kops + $0.01/GB
- GRS-HOT $0.0354/GB/mo, $0.10/10Kops
- RAGRS-COOL $0.025/GB/mo, $0.20/10Kops + $0.01/GB
- RAGRS-HOST $0.0442/GB/mo, $0.10/10Kops
- Standard - $0.022/GB, $0.004/10Kops (get)
- Infrequent Access - $0.0125/GB, $0.01/10Kops (get)
- Glacier - $0.004/GB + variable retrieval charge depending on speed
- Multi-Regional Storage $0.026/GB/mo
- Regional Storage $0.02/GB/mo
- Nearline Storage $0.01/GB/mo, $0.01/GB retrieval charge
- Coldline Storage $0.007, $0.05/GB retrieval charge
- Optional bucket versioning