Feature #13382

[keepstore] Write new blocks to appropriate storage class

Added by Tom Clegg over 3 years ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
04/02/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0
Release relationship:
Auto

Description

A PUT request can specify a storage class:

X-Keep-Storage-Classes: foo, bar

The keepstore should respond with how many replicas it was able to write for each storage class.

Keepstore should go through all volumes it has access to and attempt to record the block enough times to satisfy all requested storage classes. Once a storage class is satisfied it is removed from the list. When all storage classes are satisfied or all volumes have been checked, a response is returned: success if at least one block was written (or already exists) with any of the desired storage classes, and failure if not.

Proposed format:

X-Keep-Storage-Classes-Confirmed: foo=1, bar=1

The client (#17392) is responsible for keeping track of how many replicas have been written for each storage class. If one storage class is satisfied but not another, the satisfied one can be removed from the X-Keep-Storage-Classes list. For the client, the overall write is a success if all storage classes were satisfied, and a failure if any of the storage classes could not be satisfied.

If the keepstore server can satisfy more than one requested storage class with multiple volumes, the block should be written to all of them. If a volume has more than one storage class, it can satisfy multiple storage classes at once.

The X-Keep-Storage-Classes-Confirmed header should return the replicas per storage class. If two storage classes were satisfied then it would return "foo=1, bar=1" even if only a single replica to a single volume was actually written.

"X-Keep-Replicas-Stored" will be the sum of the actual replicas stored (for foo=1,bar=1 this could be 1 or 2 depending on whether foo and bar are the same or separate volumes).


Subtasks

Task #17482: Review 13382-write-to-storage-classResolvedTom Clegg


Related issues

Related to Arvados - Feature #11184: [Keep] Support multiple storage classesResolved

Related to Arvados - Story #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to KeepResolved05/22/2018

Related to Arvados Epics - Story #16107: Storage classesNew03/01/202109/30/2021

Has duplicate Arvados - Feature #11646: [keepstore] Support X-Keep-Desired-Classes headerClosed05/09/2017

Blocks Arvados - Story #17465: Support writing blocks to correct storage classes in Python SDKResolved06/01/2021

Associated revisions

Revision b4209cce
Added by Tom Clegg 6 months ago

Merge branch '13382-write-to-storage-class'

closes #13382

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg over 3 years ago

  • Related to Feature #11184: [Keep] Support multiple storage classes added

#2 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#4 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#5 Updated by Tom Morris over 3 years ago

  • Story points set to 1.0

#6 Updated by Tom Morris over 3 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

#7 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#8 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#9 Updated by Peter Amstutz over 3 years ago

  • Related to Story #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to Keep added

#10 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#11 Updated by Tom Clegg over 3 years ago

  • Has duplicate Feature #11646: [keepstore] Support X-Keep-Desired-Classes header added

#12 Updated by Nico C├ęsar 7 months ago

#13 Updated by Peter Amstutz 7 months ago

  • Description updated (diff)

#14 Updated by Peter Amstutz 7 months ago

  • Target version changed from Arvados Future Sprints to 2021-03-17 sprint

#15 Updated by Peter Amstutz 7 months ago

  • Target version changed from 2021-03-17 sprint to To Be Groomed
  • Description updated (diff)

#16 Updated by Peter Amstutz 7 months ago

  • Target version changed from To Be Groomed to 2021-03-17 sprint

#17 Updated by Peter Amstutz 7 months ago

  • Target version deleted (2021-03-17 sprint)

#18 Updated by Peter Amstutz 6 months ago

  • Description updated (diff)

#19 Updated by Peter Amstutz 6 months ago

  • Description updated (diff)

#20 Updated by Peter Amstutz 6 months ago

  • Story points changed from 1.0 to 3.0
  • Target version set to 2021-03-31 sprint

#21 Updated by Peter Amstutz 6 months ago

  • Assigned To set to Tom Clegg

#22 Updated by Tom Clegg 6 months ago

  • Status changed from New to In Progress

#23 Updated by Peter Amstutz 6 months ago

  • Target version changed from 2021-03-31 sprint to 2021-04-14 sprint

#24 Updated by Lucas Di Pentima 6 months ago

  • Blocks Story #17465: Support writing blocks to correct storage classes in Python SDK added

#25 Updated by Tom Clegg 6 months ago

  • Description updated (diff)

#27 Updated by Lucas Di Pentima 6 months ago

Just one minor observation:

I think up to fdd48c8, the putResult name made sense, but then as I understand it, the struct usage got a slightly different meaning. Do you think it would be clearer to name it something like put[Job|Task]?

The rest LGTM, thanks!

#28 Updated by Tom Clegg 6 months ago

Good point. How does "putProgress" sound? (I'm thinking it's just a progress tracker, doesn't have any other info about the job/task.)

13382-write-to-storage-class @ 4f33147a6937d3b22044346711b2e9c16a29f351 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2401/

#29 Updated by Tom Clegg 6 months ago

  • Status changed from In Progress to Resolved

#30 Updated by Peter Amstutz 4 months ago

  • Release set to 38

Also available in: Atom PDF