Project

General

Profile

Actions

Feature #13382

closed

[keepstore] Write new blocks to appropriate storage class

Added by Tom Clegg almost 5 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
04/02/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0
Release relationship:
Auto

Description

A PUT request can specify a storage class:

X-Keep-Storage-Classes: foo, bar

The keepstore should respond with how many replicas it was able to write for each storage class.

Keepstore should go through all volumes it has access to and attempt to record the block enough times to satisfy all requested storage classes. Once a storage class is satisfied it is removed from the list. When all storage classes are satisfied or all volumes have been checked, a response is returned: success if at least one block was written (or already exists) with any of the desired storage classes, and failure if not.

Proposed format:

X-Keep-Storage-Classes-Confirmed: foo=1, bar=1

The client (#17392) is responsible for keeping track of how many replicas have been written for each storage class. If one storage class is satisfied but not another, the satisfied one can be removed from the X-Keep-Storage-Classes list. For the client, the overall write is a success if all storage classes were satisfied, and a failure if any of the storage classes could not be satisfied.

If the keepstore server can satisfy more than one requested storage class with multiple volumes, the block should be written to all of them. If a volume has more than one storage class, it can satisfy multiple storage classes at once.

The X-Keep-Storage-Classes-Confirmed header should return the replicas per storage class. If two storage classes were satisfied then it would return "foo=1, bar=1" even if only a single replica to a single volume was actually written.

"X-Keep-Replicas-Stored" will be the sum of the actual replicas stored (for foo=1,bar=1 this could be 1 or 2 depending on whether foo and bar are the same or separate volumes).


Subtasks 1 (0 open1 closed)

Task #17482: Review 13382-write-to-storage-classResolvedTom Clegg04/02/2021

Actions

Related issues

Related to Arvados - Feature #11184: [Keep] Support multiple storage classesResolvedTom Morris

Actions
Related to Arvados - Story #13430: [arv-put] [Python] Allow caller to specify storage classes when writing data to KeepResolvedFuad Muhic05/22/2018

Actions
Related to Arvados Epics - Story #16107: Storage classesResolved03/01/202109/30/2021

Actions
Has duplicate Arvados - Feature #11646: [keepstore] Support X-Keep-Desired-Classes headerClosed05/09/2017

Actions
Blocks Arvados - Story #17465: Support writing blocks to correct storage classes in Python SDKResolvedLucas Di Pentima06/01/2021

Actions
Actions

Also available in: Atom PDF