Bug #13427

[keep-balance] Handle volumes that are mounted simultaneously by multiple servers

Added by Tom Clegg 12 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
06/06/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0
Release:
Release relationship:
Auto

Description

Example:
  • keep0 mounts vol0 (rw), vol1 (ro)
  • keep1 mounts vol1 (rw), vol0 (ro)
  • keep2 mounts vol2 (rw), vol3 (ro)
  • keep3 mounts vol3 (rw), vol2 (ro)

This setup is desirable when each block appears on only one backend volume, i.e., when the desired replication level is already provided by the backend. When a single keep server goes down, all blocks are still readable.

However, with this setup, the current keep-balance implementation will never move a block to a better rendezvous position. It sees N readonly replicas and figures there's no point making more copies on different servers: it won't be able to delete the readonly replicas, so making more replicas will result in permanent overreplication. If it pays attention to the device IDs reported by the servers, it could understand that the readonly replicas are just different views of writable replicas it sees elsewhere, and ignore them.


Subtasks

Task #13520: Review 13427-multiple-mountsResolvedWard Vandewege

Associated revisions

Revision 8f4b1051
Added by Tom Clegg 11 months ago

Merge branch '13427-multiple-mounts'

refs #13427

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Tom Clegg 11 months ago

Implementation:
  • In (*Balancer)Run(), de-duplicate devices after calling discoverMounts on all services. If the same device ID is reported by both read-only and read/write mounts, drop the read-only mounts entirely.
  • In (*Balancer)balanceBlock(), track which devices are going to be used ("wantDev") and treat this like wantMnt: don't try to use the same device twice. (When a device is mounted by multiple servers, we should prefer the one in best rendezvous position, which depends on the block -- so we can't de-duplicate these ahead of time.)

#2 Updated by Tom Clegg 11 months ago

  • Story points set to 1.0

#3 Updated by Tom Morris 11 months ago

  • Target version changed from To Be Groomed to Arvados Future Sprints

#4 Updated by Tom Morris 11 months ago

  • Target version changed from Arvados Future Sprints to 2018-06-06 Sprint

#5 Updated by Tom Morris 11 months ago

  • Assigned To set to Tom Clegg

#6 Updated by Tom Clegg 11 months ago

  • Status changed from New to In Progress
  • Target version changed from 2018-06-06 Sprint to 2018-06-20 Sprint

#7 Updated by Tom Clegg 11 months ago

#8 Updated by Tom Clegg 11 months ago

13427-multiple-mounts @ da40bd0960806df8e2799e4fb716d41ad08b169f
  • fix reported stats (count 1 replica, not 2, if it appears twice on the same device ID at different mounts)
  • de-duplicate index calls for RW-mounted devices (retrieve each index once, and apply it to all mounts with the same device ID)

#9 Updated by Tom Clegg 11 months ago

  • Status changed from In Progress to Resolved

#10 Updated by Tom Morris 9 months ago

  • Release set to 13

Also available in: Atom PDF