Feature #6611

[Data Manager] Act on pull lists for underreplicated blocks

Added by Tom Clegg over 6 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
-
Start date:
07/14/2015
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

By design, Data Manager fixed underreplicated blocks by sending "pull lists" to the keepstore servers.

Currently, Data Manager computes lists of underreplicated blocks, but doesn't send the pull lists.

Before this can be deployed:
  • data manager must send its pull lists to keep servers
  • data manager must have at least one comprehensive integration test covering a miniature version of normal operation:
    • bring up api, keepstore, and keepproxy services
    • store some collections
    • delete some blocks (using keepstore DELETE method with a suitable token) so the desired replication level is no longer met for some collections
    • run data manager
    • wait for all keepstores to finish working their trash and pull lists (by getting each keepstore's /status.json until the queues are empty and workers are idle)
    • make sure all collections' blocks are back up to their desired replication level
    • run data manager again
    • wait for all keepstores to finish working their trash and pull lists
    • make sure no more (excess) replicas have been made

Subtasks

Task #6612: Report trash and pull queue & worker status in keepstore /status.jsonNew

Task #6631: Review 6611-send-pull-listIn Progress


Related issues

Related to Arvados - Story #9162: [Block Manager] Trashes unreferenced blocks (with mtime > TTL if desired) and overreplicated blockResolved05/16/2016

History

#1 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg over 6 years ago

  • Description updated (diff)

#4 Updated by Peter Amstutz over 6 years ago

Note: there is already a preliminary 6611-send-pull-list branch that I wrote when I wrote at the same time as the "send trash list" code.

#5 Updated by Peter Amstutz almost 6 years ago

  • Target version changed from Arvados Future Sprints to 2016-05-11 sprint

#6 Updated by Tom Clegg almost 6 years ago

Fixing underreplicated blocks is a special case of rebalancing: "rebalance only the blocks that have less than desired replication."

If
  • block B has replication A ("actual"), and
  • block B should have replication D ("desired"), and
  • A < D, and
  • keep service K is one of the first D services in the probe order for block A, and
  • block B is not in service K's index
...then
  • pick some service K2 that does have block B in its index
  • add K2/B to the pull list for K.

#7 Updated by Radhika Chippada almost 6 years ago

  • Assigned To set to Radhika Chippada

#8 Updated by Radhika Chippada almost 6 years ago

  • Story points set to 2.0

#9 Updated by Brett Smith almost 6 years ago

  • Target version changed from 2016-05-11 sprint to Arvados Future Sprints

#10 Updated by Radhika Chippada over 5 years ago

  • Assigned To deleted (Radhika Chippada)

#11 Updated by Tom Clegg over 5 years ago

  • Status changed from New to Closed

#12 Updated by Tom Morris about 5 years ago

  • Target version deleted (Arvados Future Sprints)

Also available in: Atom PDF