Project

General

Profile

Recovering lost data » History » Version 3

Tom Clegg, 04/26/2019 07:58 PM

1 1 Tom Clegg
h1. Untrashing lost blocks
2
3
In some cases it is possible to recover data blocks that have been trashed by keep-balance (due to a bug or an install/config error).
4
5
If you suspect blocks have been trashed erroneously, you should immediately:
6
# On all keepstore servers: set EmptyTrashInterval to a long time like 2400h
7
# On all keepstore servers: restart keepstore
8
# Stop the keep-balance service
9
10
When you think you have corrected the underlying problem, you should:
11
# Set LostBlocksFile to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt") in your keep-balance config
12
# Start keep-balance
13
14
After keep-balance completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:
15
16
<pre><code class="bash">
17
#!/bin/bash
18
set -e
19
20
# see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml
21
token=xxxxxxx-your-system-auth-token-xxxxxxx
22
23
# all keep server hostnames
24
hosts=(keep0 keep1 keep2 keep3 keep4 keep5)
25
26 3 Tom Clegg
while read hash pdhs; do
27 1 Tom Clegg
    echo "${hash}"
28
    for h in ${hosts[@]}; do
29 2 Nico César
        if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then
30 1 Tom Clegg
            echo "${hash} ok ${host}"
31
        fi
32
    done
33
done < /tmp/keep-balance-lost-blocks.txt
34
</code></pre>