Arvados » Arvados Epics

Requirements and high level design

• Each keep volume is assigned one or more storage classes
• The system asynchronously copies/moves blocks between Keep volumes to fulfill the desired storage classes for each block.

When uploading a block, the Python and Go SDKs support passing a list of storage classes to influence placement of the data block.
Each data block will be stored on volumes that fulfill the desired storage classes.

• A block that has two or more storage classes may be fulfilled by a single volume that fulfills all storage classes, or multiple volumes for each storage class.
• [#17349] The “replicas_desired” field expresses the number of replicas per storage class.
  • A single volume can be configured to count as multiple replicas (existing behavior)
  • [#17350 for keep-balance] A block assigned to two storage classes with N=2 replicas could have 2, 3 or 4 actual copies. For example, a block requesting storage classes A and B could be written to two volumes with storage classes “A and B”, or two “A” volumes and two “B” volumes, or “A”, “B” and “A,B”.
  • If sufficient replicas cannot be written for each storage class during upload, it is a fatal error (existing behavior)

• Storage classes are advisory and do not guarantee that a block will only be stored on a certain class or will never be stored on a certain class. Some circumstances where blocks may not be stored on only the requested storage class:
  • User has specified an impossible situation, such as changing a collection to a storage class that isn’t fulfilled by any volume
  • The same block may be referenced by multiple collections with different storage classes
  • A storage class for a collection may have been changed but the system has not caught up to it yet
  • In these cases the block will remain on its original storage volume
  • There will be a reporting tool (keep-balance or other) that reports when there is a mismatch between the desired and actual storage classes for a block. The tool will also provide a way to report which blocks (associated collections) on a certain storage class also exist on a different storage class.

• When writing a block, the desired storage classes are passed in the keep service request
• [#17389] Keepproxy will understand storage classes and forwards blocks to the appropriate keepstore service.
• When reading a collection, the Python and Go SDK support passing a list of storage classes to inform block lookup.
• There is always a “default” storage class. It is an error if there is not at least one volume with the “default” class.
  • If an upload doesn’t specify a storage class it will use the ‘default’ storage class
• Container and container request records gain a field specifying the storage class of the output collection.
• [#17351][#17390] The following Arvados component swill gain support for specifying storage classes for data upload: arv-put, arvados-cwl-runner, crunch-run
  • When writing a block, the desired storage classes are passed in the keep service request
  • Keepproxy will understand storage classes and forwards blocks to the appropriate keepstore service.
  • When reading a collection, the Python and Go SDK support passing a list of storage classes to inform block lookup.
  • There is always a “default” storage class. It is an error if there is not at least one volume with the “default” class.
  • If an upload doesn’t specify a storage class it will use the ‘default’ storage class
  • Container and container request records gain a field specifying the storage class of the output collection.
• Workbench 2 collection view will display the storage class of the collection.
• Work will be tested with unit, functional and integration testing using Amazon S3