Project

General

Profile

Actions

Feature #17967

closed

Prioritize reads from different storage classes

Added by Peter Amstutz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
08/05/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Add a list of storage classes

StorageClasses:
  default: {Priority: 100, DefaultWrite: true}
  foo: {Priority: 50, DefaultWrite: false}
  bar: {Priority: 20, DefaultWrite: false}

Higher priority has higher precedence.

Validate the storage classes of each volume list against the main list.

Also includes adding an exported set of possibly multiple "default" storage classes.

Use priority to determine order to check volumes for blocks.


Subtasks 2 (0 open2 closed)

Task #17972: Review 17967-storage-classes-configResolvedPeter Amstutz08/06/2021

Actions
Task #17978: Add storage class default/priority fields to config + exported configResolvedTom Clegg08/05/2021

Actions

Related issues

Related to Arvados Epics - Story #16107: Storage classesResolved03/01/202109/30/2021

Actions
Blocks Arvados - Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overriddenResolvedLucas Di Pentima08/19/2021

Actions
Actions #1

Updated by Peter Amstutz over 1 year ago

Actions #2

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Category set to Keep
Actions #4

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz over 1 year ago

  • Assigned To set to Tom Clegg
Actions #7

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz over 1 year ago

  • Blocks Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overridden added
Actions #10

Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress
Actions #11

Updated by Tom Clegg over 1 year ago

It seemed reasonable to add the whole StorageClasses config section (with Default for #17696 as well as Priority for this issue). Then I forgot #17696 was already assigned to someone else and implemented the Default behavior as well.

17967-storage-classes-config @ 13b280ea288fd08e8c77300b087037635d739eaa -- developer-run-tests: #2621

  • add config section with default behavior as described in #17696
  • try higher-priority volumes (i.e., volumes with higher-priority storage classes) first when reading
  • change keepstore behavior to write to the configured default storage classes when the client didn't specify any (previously was "disregard storage classes" in that case) -- I figured this should make the SDK side much easier
  • change arv-put behavior to not specify storage classes for newly written collections/blocks if none were specified by the caller (previously was ["default"])
  • TODO: update admin/storage-classes.html
Actions #12

Updated by Tom Clegg over 1 year ago

17967-storage-classes-config @ 2de3b2e44f2196c227c7bbd57b211d1095e59aad
  • update docs
Actions #13

Updated by Peter Amstutz over 1 year ago

      # If you use multiple storage classes, specify them here, using
      # the storage class name as the key (in place of "SAMPLE" in
      # this sample entry).

This comment doesn't explain what a storage class is, it should either be extended a bit ("Storage classes enable you to control which volumes should be used to store the blocks of a given collection") and/or link to the documentation.

Need to mention the config changes in doc/admin/upgrading.html.textile.liquid

There's some changes to the API server behavior, perhaps that should get a test?

Actions #14

Updated by Tom Clegg over 1 year ago

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- developer-run-tests: #2625
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes
Actions #15

Updated by Peter Amstutz over 1 year ago

Tom Clegg wrote:

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- developer-run-tests: #2625
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes

This LGTM

Actions #16

Updated by Tom Clegg over 1 year ago

  • Status changed from In Progress to Resolved
Actions #17

Updated by Peter Amstutz about 1 year ago

  • Release set to 42
Actions

Also available in: Atom PDF