Feature #11184

[Keep] Support multiple storage classes

Added by Tom Morris about 1 year ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/05/2017
Due date:
% Done:

0%

Estimated time:
Story points:
3.0

Description

As an Arvados system administrator, I want to take advantage of the cool/cold storage classes offered by cloud vendors.

This involves designating desired storage class(es) in some way (collection, all collections in a project, etc) as well as a way to migrate between storage classes.

Keep storage classes

Overview

  • Each keep volume offers one or more storage classes (the default is just the "default" class).
  • Each collection has one or more desired storage classes (the default is just the "default" class).
  • When writing, clients may specify one or more required storage classes; if not, the required class is "default". A keepstore server will only write the data on a volume that offers all of the required classes.
  • Keep-balance moves data to volumes that have the desired attributes and updates collection records to reflect the storage classes currently satisfied by all blocks in the collection (much like replication level, these are not necessarily equal to the desired classes).

If overlapping collections (i.e., with common data blocks) request different storage classes, keep-balance will maintain multiple copies of the common blocks if necessary to satisfy all collections' requirements.

Initial implementation

Simplifying restrictions:
  • No client side support.
  • No keepstore support for writing data to a given storage class.
  • API server configuration specifies the set of classes that can be requested (in addition to "default", which is always available).
  • Keepstore configuration specifies the set of classes offered by each volume.

Related issues

Related to Arvados - Story #7929: [SDKs] PySDK KeepClient considers volume IDs when replicatingNew2015-12-02

Related to Arvados - Story #7930: [SDKs] GoSDK KeepClient considers volume IDs when replicatingNew2015-12-02

Related to Arvados - Story #7931: [keep-balance] Count block replication by volume IDsResolved2015-12-02

Related to Arvados - Feature #11644: [keepstore] mount-oriented APIsResolved2017-05-09

Related to Arvados - Feature #13382: [keepstore] Write new blocks to appropriate storage classNew

Blocked by Arvados - Feature #11645: [keepstore] Add "StorageClasses" field to volume configResolved2017-05-09

Blocked by Arvados - Story #12707: [API] Add columns for desired/actual storage classes for each collectionResolved2018-02-22

Blocked by Arvados - Story #12708: [keep-balance] Move blocks to satisfy storage_classes_desired, and update storage_classes_confirmed*In Progress2018-04-19

History

#1 Updated by Tom Morris about 1 year ago

  • Assigned To set to Tom Clegg

#2 Updated by Tom Clegg 12 months ago

  • Description updated (diff)

#3 Updated by Tom Morris 11 months ago

  • Assigned To changed from Tom Clegg to Tom Morris

#4 Updated by Tom Morris 11 months ago

  • Description updated (diff)

#5 Updated by Tom Clegg 7 months ago

suggest
  1. specify desired functionality (user can push X buttons, result is Y)
  2. enumerate features/improvements needed to achieve desired functionality, and define any new APIs needed
  3. estimate points (or maybe split into separate stories if needed)

#7 Updated by Tom Morris 7 months ago

  • Description updated (diff)

#8 Updated by Tom Morris 7 months ago

  • Story points set to 3.0

#9 Updated by Peter Amstutz 5 months ago

  • Target version changed from Arvados Future Sprints to To Be Groomed

#10 Updated by Peter Amstutz 5 months ago

  • Assigned To changed from Tom Morris to Tom Clegg

#11 Updated by Tom Clegg 5 months ago

  • Blocked by Feature #11645: [keepstore] Add "StorageClasses" field to volume config added

#12 Updated by Tom Clegg 5 months ago

  • Subject changed from [Keep] Support multiple storage tiers to [Keep] Support multiple storage classes
  • Description updated (diff)

Pricing examples (moved from description)

Microsoft Azure (pricing at 50-500 TB level)

  • LRS-COOL $0.01/GB/mo, $0.01/10Kops + $0.01GB
  • LRS-HOT $0.0177/GB/mo, $0.05/10Kops
  • GRS-COOL $0.02/GB/mo, $0.20/10Kops + $0.01/GB
  • GRS-HOT $0.0354/GB/mo, $0.10/10Kops
  • RAGRS-COOL $0.025/GB/mo, $0.20/10Kops + $0.01/GB
  • RAGRS-HOST $0.0442/GB/mo, $0.10/10Kops
Amazon S3 (pricing at 50-500TB level)
  • Standard - $0.022/GB, $0.004/10Kops (get)
  • Infrequent Access - $0.0125/GB, $0.01/10Kops (get)
  • Glacier - $0.004/GB + variable retrieval charge depending on speed
Google
  • Multi-Regional Storage $0.026/GB/mo
  • Regional Storage $0.02/GB/mo
  • Nearline Storage $0.01/GB/mo, $0.01/GB retrieval charge
  • Coldline Storage $0.007, $0.05/GB retrieval charge
  • Optional bucket versioning

#13 Updated by Tom Clegg 5 months ago

  • Blocked by Story #12707: [API] Add columns for desired/actual storage classes for each collection added

#14 Updated by Tom Clegg 5 months ago

  • Blocked by Story #12708: [keep-balance] Move blocks to satisfy storage_classes_desired, and update storage_classes_confirmed* added

#15 Updated by Tom Clegg 5 months ago

  • Description updated (diff)

#16 Updated by Tom Clegg 5 months ago

  • Related to Story #7929: [SDKs] PySDK KeepClient considers volume IDs when replicating added

#17 Updated by Tom Clegg 5 months ago

  • Related to Story #7930: [SDKs] GoSDK KeepClient considers volume IDs when replicating added

#18 Updated by Tom Clegg 5 months ago

  • Related to Story #7931: [keep-balance] Count block replication by volume IDs added

#19 Updated by Tom Clegg 5 months ago

  • Related to Story #7932: [Keep] keepproxy aggregates and reports volume IDs from Keepstores added

#20 Updated by Tom Morris about 2 months ago

  • Target version changed from To Be Groomed to 2018-03-14 Sprint

#21 Updated by Tom Clegg about 1 month ago

  • Target version changed from 2018-03-14 Sprint to 2018-03-28 Sprint

#22 Updated by Tom Morris about 1 month ago

  • Target version changed from 2018-03-28 Sprint to Arvados Future Sprints

#23 Updated by Tom Clegg about 1 month ago

#24 Updated by Tom Morris 14 days ago

  • Related to deleted (Story #7932: [Keep] keepproxy aggregates and reports volume IDs from Keepstores)

#25 Updated by Tom Clegg about 15 hours ago

  • Related to Feature #13382: [keepstore] Write new blocks to appropriate storage class added

Also available in: Atom PDF