Arvados

In order to make correct rendezvous decisions, when a volume is mounted at multiple keepstore servers, keep-balance internally pictures a replica on each mount.

This has the unintended consequence that each trash list entry is sent to all servers that mount the relevant volume, instead of just one.

In most cases this is harmless: one of the servers will win the race to trash the block, and the rest will do nothing because the block is already trashed.

However, when using the S3 driver, the trash strategy has a race detection/recovery feature that is triggered when multiple keepstore processes try to trash a block at the same time, and the result is that the block is not trashed.

To fix this, keep-balance should send each trash list item to only one (randomly selected) server that mounts the relevant volume, not all of them.