Recovering Lost Data¶

Untrashing lost blocks¶

In some cases it is possible to recover data blocks that have been trashed by keep-balance (due to a bug like #15148, or an install/config error).

If you suspect blocks have been trashed erroneously, you should immediately:

1. On all keepstore servers: set EmptyTrashInterval to a long time like 2400h
2. On all keepstore servers: restart keepstore
3. Stop the keep-balance service

When you think you have corrected the underlying problem, you should:

1. Set LostBlocksFile to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt") in your keep-balance config
2. Start keep-balance

After keep-balance completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:

#!/bin/bash
set -e

# see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml
token=xxxxxxx-your-system-auth-token-xxxxxxx

# all keep server hostnames
hosts=(keep0 keep1 keep2 keep3 keep4 keep5)

while read hash pdhs; do
    echo "${hash}" 
    for h in ${hosts[@]}; do
        if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then
            echo "${hash} ok ${host}" 
        fi
    done
done < /tmp/keep-balance-lost-blocks.txt

Obviously this could be improved upon with increased parallelism for large scale tasks, if needed. Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.

Regenerating missing blocks¶

For blocks which were trashed long enough that they've been deleted, it's possible to regenerate them by rerunning the workflows which generated them. To do this, the process is:

1. Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all our, so they're unlikely to contain any useful data)
2. Resubmit any container requests that you want to regenerate the output collections for

There's tool script that can be used to generate a report to help with this task in the Arvados repository at arvados/tools/keep-xref/keep-xref.py