Feature #17967

Prioritize reads from different storage classes

Added by Peter Amstutz 3 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
08/05/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

Add a list of storage classes

StorageClasses:
  default: {Priority: 100, DefaultWrite: true}
  foo: {Priority: 50, DefaultWrite: false}
  bar: {Priority: 20, DefaultWrite: false}

Higher priority has higher precedence.

Validate the storage classes of each volume list against the main list.

Also includes adding an exported set of possibly multiple "default" storage classes.

Use priority to determine order to check volumes for blocks.


Subtasks

Task #17972: Review 17967-storage-classes-configResolvedPeter Amstutz

Task #17978: Add storage class default/priority fields to config + exported configResolvedTom Clegg


Related issues

Related to Arvados Epics - Story #16107: Storage classesIn Progress03/01/202109/30/2021

Blocks Arvados - Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overriddenResolved08/19/2021

Associated revisions

Revision 818e9974
Added by Tom Clegg 2 months ago

Merge branch '17967-storage-classes-config' into main

closes #17967

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 3 months ago

#2 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 3 months ago

  • Category set to Keep

#4 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 2 months ago

  • Assigned To set to Tom Clegg

#7 Updated by Peter Amstutz 2 months ago

  • Description updated (diff)

#8 Updated by Peter Amstutz 2 months ago

  • Description updated (diff)

#9 Updated by Peter Amstutz 2 months ago

  • Blocks Feature #17696: Exported config has default storage class(es), SDKs use the configured default storage class if not overridden added

#10 Updated by Tom Clegg 2 months ago

  • Status changed from New to In Progress

#11 Updated by Tom Clegg 2 months ago

It seemed reasonable to add the whole StorageClasses config section (with Default for #17696 as well as Priority for this issue). Then I forgot #17696 was already assigned to someone else and implemented the Default behavior as well.

17967-storage-classes-config @ 13b280ea288fd08e8c77300b087037635d739eaa -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2621/

  • add config section with default behavior as described in #17696
  • try higher-priority volumes (i.e., volumes with higher-priority storage classes) first when reading
  • change keepstore behavior to write to the configured default storage classes when the client didn't specify any (previously was "disregard storage classes" in that case) -- I figured this should make the SDK side much easier
  • change arv-put behavior to not specify storage classes for newly written collections/blocks if none were specified by the caller (previously was ["default"])
  • TODO: update admin/storage-classes.html

#12 Updated by Tom Clegg 2 months ago

17967-storage-classes-config @ 2de3b2e44f2196c227c7bbd57b211d1095e59aad
  • update docs

#13 Updated by Peter Amstutz 2 months ago

      # If you use multiple storage classes, specify them here, using
      # the storage class name as the key (in place of "SAMPLE" in
      # this sample entry).

This comment doesn't explain what a storage class is, it should either be extended a bit ("Storage classes enable you to control which volumes should be used to store the blocks of a given collection") and/or link to the documentation.

Need to mention the config changes in doc/admin/upgrading.html.textile.liquid

There's some changes to the API server behavior, perhaps that should get a test?

#14 Updated by Tom Clegg 2 months ago

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2625/
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes

#15 Updated by Peter Amstutz 2 months ago

Tom Clegg wrote:

17967-storage-classes-config @ 7860f93d77e8690ba90e839b52e2886a08466773 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2625/
  • add doc link to config file
  • add upgrade note about adding StorageClasses section
  • add/update railsapi tests for default storage classes

This LGTM

#16 Updated by Tom Clegg 2 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF