Project

General

Profile

Actions

Feature #17967

closed

Prioritize reads from different storage classes

Added by Peter Amstutz over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
-
Release relationship:
Auto

Description

Add a list of storage classes

StorageClasses:
  default: {Priority: 100, DefaultWrite: true}
  foo: {Priority: 50, DefaultWrite: false}
  bar: {Priority: 20, DefaultWrite: false}

Higher priority has higher precedence.

Validate the storage classes of each volume list against the main list.

Also includes adding an exported set of possibly multiple "default" storage classes.

Use priority to determine order to check volumes for blocks.


Subtasks 2 (0 open2 closed)

Task #17972: Review 17967-storage-classes-configResolvedPeter Amstutz08/06/2021Actions
Task #17978: Add storage class default/priority fields to config + exported configResolvedTom Clegg08/05/2021Actions

Related issues

Related to Arvados Epics - Idea #16107: Storage classesResolved03/01/202109/30/2021Actions
Blocks Arvados - Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overriddenResolvedLucas Di Pentima08/19/2021Actions
Actions #1

Updated by Peter Amstutz over 2 years ago

Actions #2

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz over 2 years ago

  • Category set to Keep
Actions #4

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz over 2 years ago

  • Assigned To set to Tom Clegg
Actions #7

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #8

Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz over 2 years ago

  • Blocks Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overridden added
Actions #10

Updated by Tom Clegg over 2 years ago

  • Status changed from New to In Progress
Actions #11

Updated by Tom Clegg over 2 years ago

It seemed reasonable to add the whole StorageClasses config section (with Default for #17696 as well as Priority for this issue). Then I forgot #17696 was already assigned to someone else and implemented the Default behavior as well.

17967-storage-classes-config @ 13b280ea288fd08e8c77300b087037635d739eaa -- developer-run-tests: #2621

  • add config section with default behavior as described in #17696
  • try higher-priority volumes (i.e., volumes with higher-priority storage classes) first when reading
  • change keepstore behavior to write to the configured default storage classes when the client didn't specify any (previously was "disregard storage classes" in that case) -- I figured this should make the SDK side much easier
  • change arv-put behavior to not specify storage classes for newly written collections/blocks if none were specified by the caller (previously was ["default"])
  • TODO: update admin/storage-classes.html
Actions #12

Updated by Tom Clegg over 2 years ago

17967-storage-classes-config @ 2de3b2e44f2196c227c7bbd57b211d1095e59aad
  • update docs
Actions #13

Updated by Peter Amstutz over 2 years ago

      # If you use multiple storage classes, specify them here, using
      # the storage class name as the key (in place of "SAMPLE" in
      # this sample entry).

This comment doesn't explain what a storage class is, it should either be extended a bit ("Storage classes enable you to control which volumes should be used to store the blocks of a given collection") and/or link to the documentation.

Need to mention the config changes in doc/admin/upgrading.html.textile.liquid

There's some changes to the API server behavior, perhaps that should get a test?

Actions #14

Updated by Tom Clegg over 2 years ago

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- developer-run-tests: #2625
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes
Actions #15

Updated by Peter Amstutz over 2 years ago

Tom Clegg wrote:

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- developer-run-tests: #2625
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes

This LGTM

Actions #16

Updated by Tom Clegg over 2 years ago

  • Status changed from In Progress to Resolved
Actions #17

Updated by Peter Amstutz over 2 years ago

  • Release set to 42
Actions

Also available in: Atom PDF