Project

General

Profile

Actions

Feature #11184

closed

[Keep] Support multiple storage classes

Added by Tom Morris over 7 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Morris
Category:
-
Target version:
-
Story points:
-

Description

As an Arvados system administrator, I want to take advantage of the cool/cold storage classes offered by cloud vendors.

This involves designating desired storage class(es) in some way (collection, all collections in a project, etc) as well as a way to migrate between storage classes.

Keep storage classes

Overview

  • Each keep volume offers one or more storage classes (the default is just the "default" class).
  • Each collection has one or more desired storage classes (the default is just the "default" class).
  • When writing, clients may specify one or more required storage classes; if not, the required class is "default". A keepstore server will only write the data on a volume that offers all of the required classes.
  • Keep-balance moves data to volumes that have the desired attributes and updates collection records to reflect the storage classes currently satisfied by all blocks in the collection (much like replication level, these are not necessarily equal to the desired classes).

If overlapping collections (i.e., with common data blocks) request different storage classes, keep-balance will maintain multiple copies of the common blocks if necessary to satisfy all collections' requirements.

Simplifying restrictions in initial implementation (#12708):
  • No client side support.
  • No keepstore support for writing data to a given storage class.
  • API server configuration specifies the set of classes that can be requested (in addition to "default", which is always available).
  • Keepstore configuration specifies the set of classes offered by each volume.

Related issues

Related to Arvados - Idea #7931: [keep-balance] Count block replication by volume IDsResolvedTom Clegg12/02/2015Actions
Related to Arvados - Feature #11644: [keepstore] mount-oriented APIsResolvedTom Clegg05/09/2017Actions
Related to Arvados - Feature #13382: [keepstore] Write new blocks to appropriate storage classResolvedTom Clegg04/02/2021Actions
Related to Arvados - Feature #17392: Support writing blocks to correct storage classes in Go SDKResolvedTom Clegg04/12/2021Actions
Related to Arvados Epics - Idea #16107: Storage classesResolved03/01/202109/30/2021Actions
Blocked by Arvados - Feature #11645: [keepstore] Add "StorageClasses" field to volume configResolvedPeter Amstutz05/09/2017Actions
Blocked by Arvados - Idea #12707: [API] Add columns for desired/actual storage classes for each collectionResolvedLucas Di Pentima02/22/2018Actions
Blocked by Arvados - Idea #12708: [keep-balance] Move blocks to satisfy storage_classes_desiredResolvedTom Clegg04/19/2018Actions
Blocked by Arvados - Feature #13431: [keepproxy] [GoSDK] Propagate desired storage classes in PUT request headersResolvedLucas Di Pentima06/05/2018Actions
Blocked by Arvados - Idea #13429: [API] [arvados-cwl-runner] Save workflow outputs to desired storage classesClosedFuad Muhic06/04/2018Actions
Blocked by Arvados - Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to KeepResolvedFuad Muhic05/22/2018Actions
Actions #1

Updated by Tom Morris over 7 years ago

  • Assigned To set to Tom Clegg
Actions #2

Updated by Tom Clegg over 7 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Morris over 7 years ago

  • Assigned To changed from Tom Clegg to Tom Morris
Actions #4

Updated by Tom Morris over 7 years ago

  • Description updated (diff)
Actions #5

Updated by Tom Clegg about 7 years ago

suggest
  1. specify desired functionality (user can push X buttons, result is Y)
  2. enumerate features/improvements needed to achieve desired functionality, and define any new APIs needed
  3. estimate points (or maybe split into separate stories if needed)
Actions #7

Updated by Tom Morris about 7 years ago

  • Description updated (diff)
Actions #8

Updated by Tom Morris about 7 years ago

  • Story points set to 3.0
Actions #9

Updated by Peter Amstutz almost 7 years ago

  • Target version changed from Arvados Future Sprints to To Be Groomed
Actions #10

Updated by Peter Amstutz almost 7 years ago

  • Assigned To changed from Tom Morris to Tom Clegg
Actions #11

Updated by Tom Clegg almost 7 years ago

  • Blocked by Feature #11645: [keepstore] Add "StorageClasses" field to volume config added
Actions #12

Updated by Tom Clegg almost 7 years ago

  • Subject changed from [Keep] Support multiple storage tiers to [Keep] Support multiple storage classes
  • Description updated (diff)

Pricing examples (moved from description)

Microsoft Azure (pricing at 50-500 TB level)

  • LRS-COOL $0.01/GB/mo, $0.01/10Kops + $0.01GB
  • LRS-HOT $0.0177/GB/mo, $0.05/10Kops
  • GRS-COOL $0.02/GB/mo, $0.20/10Kops + $0.01/GB
  • GRS-HOT $0.0354/GB/mo, $0.10/10Kops
  • RAGRS-COOL $0.025/GB/mo, $0.20/10Kops + $0.01/GB
  • RAGRS-HOST $0.0442/GB/mo, $0.10/10Kops
Amazon S3 (pricing at 50-500TB level)
  • Standard - $0.022/GB, $0.004/10Kops (get)
  • Infrequent Access - $0.0125/GB, $0.01/10Kops (get)
  • Glacier - $0.004/GB + variable retrieval charge depending on speed
Google
  • Multi-Regional Storage $0.026/GB/mo
  • Regional Storage $0.02/GB/mo
  • Nearline Storage $0.01/GB/mo, $0.01/GB retrieval charge
  • Coldline Storage $0.007, $0.05/GB retrieval charge
  • Optional bucket versioning
Actions #13

Updated by Tom Clegg almost 7 years ago

  • Blocked by Idea #12707: [API] Add columns for desired/actual storage classes for each collection added
Actions #14

Updated by Tom Clegg almost 7 years ago

  • Blocked by Idea #12708: [keep-balance] Move blocks to satisfy storage_classes_desired added
Actions #15

Updated by Tom Clegg almost 7 years ago

  • Description updated (diff)
Actions #16

Updated by Tom Clegg almost 7 years ago

  • Related to Idea #7929: [SDKs] PySDK KeepClient considers volume IDs when replicating added
Actions #17

Updated by Tom Clegg almost 7 years ago

  • Related to Idea #7930: [SDKs] GoSDK KeepClient considers volume IDs when replicating added
Actions #18

Updated by Tom Clegg almost 7 years ago

  • Related to Idea #7931: [keep-balance] Count block replication by volume IDs added
Actions #19

Updated by Tom Clegg almost 7 years ago

  • Related to Idea #7932: [Keep] keepproxy aggregates and reports volume IDs from Keepstores added
Actions #20

Updated by Tom Morris over 6 years ago

  • Target version changed from To Be Groomed to 2018-03-14 Sprint
Actions #21

Updated by Tom Clegg over 6 years ago

  • Target version changed from 2018-03-14 Sprint to 2018-03-28 Sprint
Actions #22

Updated by Tom Morris over 6 years ago

  • Target version changed from 2018-03-28 Sprint to Arvados Future Sprints
Actions #23

Updated by Tom Clegg over 6 years ago

Actions #24

Updated by Tom Morris over 6 years ago

  • Related to deleted (Idea #7932: [Keep] keepproxy aggregates and reports volume IDs from Keepstores)
Actions #25

Updated by Tom Clegg over 6 years ago

  • Related to Feature #13382: [keepstore] Write new blocks to appropriate storage class added
Actions #26

Updated by Tom Morris over 6 years ago

  • Related to Idea #13429: [API] [arvados-cwl-runner] Save workflow outputs to desired storage classes added
Actions #27

Updated by Tom Morris over 6 years ago

  • Related to Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to Keep added
Actions #28

Updated by Tom Clegg over 6 years ago

  • Blocked by Feature #13431: [keepproxy] [GoSDK] Propagate desired storage classes in PUT request headers added
Actions #29

Updated by Tom Clegg over 6 years ago

  • Related to deleted (Idea #7929: [SDKs] PySDK KeepClient considers volume IDs when replicating)
Actions #30

Updated by Tom Clegg over 6 years ago

  • Related to deleted (Idea #7930: [SDKs] GoSDK KeepClient considers volume IDs when replicating)
Actions #31

Updated by Tom Clegg over 6 years ago

  • Related to deleted (Idea #13429: [API] [arvados-cwl-runner] Save workflow outputs to desired storage classes)
Actions #32

Updated by Tom Clegg over 6 years ago

  • Blocked by Idea #13429: [API] [arvados-cwl-runner] Save workflow outputs to desired storage classes added
Actions #33

Updated by Tom Clegg over 6 years ago

  • Related to deleted (Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to Keep)
Actions #34

Updated by Tom Clegg over 6 years ago

  • Blocked by Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to Keep added
Actions #35

Updated by Tom Clegg over 6 years ago

  • Description updated (diff)
Actions #36

Updated by Tom Morris over 6 years ago

  • Target version changed from Arvados Future Sprints to 2018-06-20 Sprint
Actions #37

Updated by Tom Morris over 6 years ago

  • Assigned To changed from Tom Clegg to Tom Morris
  • Story points deleted (3.0)
Actions #38

Updated by Tom Morris over 6 years ago

  • Status changed from New to In Progress
Actions #39

Updated by Tom Morris over 6 years ago

  • Target version changed from 2018-06-20 Sprint to Arvados Future Sprints
Actions #40

Updated by Tom Clegg over 3 years ago

  • Related to Feature #17392: Support writing blocks to correct storage classes in Go SDK added
Actions #41

Updated by Tom Clegg over 3 years ago

Actions #42

Updated by Peter Amstutz over 3 years ago

  • Status changed from In Progress to Resolved
Actions #43

Updated by Ward Vandewege over 3 years ago

  • Target version changed from Arvados Future Sprints to 0
Actions #44

Updated by Ward Vandewege over 3 years ago

  • Target version deleted (0)
Actions

Also available in: Atom PDF