Project

General

Profile

Actions

Feature #13382

closed

[keepstore] Write new blocks to appropriate storage class

Added by Tom Clegg almost 6 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
3.0
Release relationship:
Auto

Description

A PUT request can specify a storage class:

X-Keep-Storage-Classes: foo, bar

The keepstore should respond with how many replicas it was able to write for each storage class.

Keepstore should go through all volumes it has access to and attempt to record the block enough times to satisfy all requested storage classes. Once a storage class is satisfied it is removed from the list. When all storage classes are satisfied or all volumes have been checked, a response is returned: success if at least one block was written (or already exists) with any of the desired storage classes, and failure if not.

Proposed format:

X-Keep-Storage-Classes-Confirmed: foo=1, bar=1

The client (#17392) is responsible for keeping track of how many replicas have been written for each storage class. If one storage class is satisfied but not another, the satisfied one can be removed from the X-Keep-Storage-Classes list. For the client, the overall write is a success if all storage classes were satisfied, and a failure if any of the storage classes could not be satisfied.

If the keepstore server can satisfy more than one requested storage class with multiple volumes, the block should be written to all of them. If a volume has more than one storage class, it can satisfy multiple storage classes at once.

The X-Keep-Storage-Classes-Confirmed header should return the replicas per storage class. If two storage classes were satisfied then it would return "foo=1, bar=1" even if only a single replica to a single volume was actually written.

"X-Keep-Replicas-Stored" will be the sum of the actual replicas stored (for foo=1,bar=1 this could be 1 or 2 depending on whether foo and bar are the same or separate volumes).


Subtasks 1 (0 open1 closed)

Task #17482: Review 13382-write-to-storage-classResolvedTom Clegg04/02/2021Actions

Related issues

Related to Arvados - Feature #11184: [Keep] Support multiple storage classesResolvedTom MorrisActions
Related to Arvados - Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to KeepResolvedFuad Muhic05/22/2018Actions
Related to Arvados Epics - Idea #16107: Storage classesResolved03/01/202109/30/2021Actions
Has duplicate Arvados - Feature #11646: [keepstore] Support X-Keep-Desired-Classes headerClosed05/09/2017Actions
Blocks Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKResolvedLucas Di Pentima06/01/2021Actions
Actions #1

Updated by Tom Clegg almost 6 years ago

  • Related to Feature #11184: [Keep] Support multiple storage classes added
Actions #2

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #3

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #4

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #5

Updated by Tom Morris almost 6 years ago

  • Story points set to 1.0
Actions #6

Updated by Tom Morris almost 6 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
Actions #7

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #8

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #9

Updated by Peter Amstutz almost 6 years ago

  • Related to Idea #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to Keep added
Actions #10

Updated by Tom Clegg almost 6 years ago

  • Description updated (diff)
Actions #11

Updated by Tom Clegg almost 6 years ago

  • Has duplicate Feature #11646: [keepstore] Support X-Keep-Desired-Classes header added
Actions #12

Updated by Nico César about 3 years ago

Actions #13

Updated by Peter Amstutz about 3 years ago

  • Description updated (diff)
Actions #14

Updated by Peter Amstutz about 3 years ago

  • Target version changed from Arvados Future Sprints to 2021-03-17 sprint
Actions #15

Updated by Peter Amstutz about 3 years ago

  • Target version changed from 2021-03-17 sprint to To Be Groomed
  • Description updated (diff)
Actions #16

Updated by Peter Amstutz about 3 years ago

  • Target version changed from To Be Groomed to 2021-03-17 sprint
Actions #17

Updated by Peter Amstutz about 3 years ago

  • Target version deleted (2021-03-17 sprint)
Actions #18

Updated by Peter Amstutz about 3 years ago

  • Description updated (diff)
Actions #19

Updated by Peter Amstutz about 3 years ago

  • Description updated (diff)
Actions #20

Updated by Peter Amstutz about 3 years ago

  • Story points changed from 1.0 to 3.0
  • Target version set to 2021-03-31 sprint
Actions #21

Updated by Peter Amstutz about 3 years ago

  • Assigned To set to Tom Clegg
Actions #22

Updated by Tom Clegg almost 3 years ago

  • Status changed from New to In Progress
Actions #23

Updated by Peter Amstutz almost 3 years ago

  • Target version changed from 2021-03-31 sprint to 2021-04-14 sprint
Actions #24

Updated by Lucas Di Pentima almost 3 years ago

  • Blocks Idea #17465: Support writing blocks to correct storage classes in Python SDK added
Actions #25

Updated by Tom Clegg almost 3 years ago

  • Description updated (diff)
Actions #27

Updated by Lucas Di Pentima almost 3 years ago

Just one minor observation:

I think up to fdd48c8, the putResult name made sense, but then as I understand it, the struct usage got a slightly different meaning. Do you think it would be clearer to name it something like put[Job|Task]?

The rest LGTM, thanks!

Actions #28

Updated by Tom Clegg almost 3 years ago

Good point. How does "putProgress" sound? (I'm thinking it's just a progress tracker, doesn't have any other info about the job/task.)

13382-write-to-storage-class @ 4f33147a6937d3b22044346711b2e9c16a29f351 -- developer-run-tests: #2401

Actions #29

Updated by Tom Clegg almost 3 years ago

  • Status changed from In Progress to Resolved
Actions #30

Updated by Peter Amstutz almost 3 years ago

  • Release set to 38
Actions

Also available in: Atom PDF