Arvados

h1. Recovering Lost Data

h2. Untrashing lost blocks

In some cases it is possible to recover data blocks that have been trashed by keep-balance (due to a bug like #15148, or an install/config error).

If you suspect blocks have been trashed erroneously, you should immediately:

# On all keepstore servers: set EmptyTrashInterval to a long time like 2400h

# On all keepstore servers: restart keepstore

# Stop the keep-balance service

When you think you have corrected the underlying problem, you should:

# Set LostBlocksFile to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt") in your keep-balance config (Arvados v1.3.3 or later)

# Start keep-balance

After keep-balance completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this:

<pre><code class="bash">

#!/bin/bash

set -e

# see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml

token=xxxxxxx-your-system-auth-token-xxxxxxx

# all keep server hostnames

hosts=(keep0 keep1 keep2 keep3 keep4 keep5)

while read hash pdhs; do

    echo "${hash}"

    for h in ${hosts[@]}; do

        if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then

            echo "${hash} ok ${host}"

        fi

    done

done < /tmp/keep-balance-lost-blocks.txt

</code></pre>

Obviously this could be improved upon with increased parallelism for large scale tasks, if needed. Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered.

h2. Regenerating missing blocks

For blocks which were trashed long enough that they've been deleted, it's possible to regenerate them by rerunning the workflows which generated them. To do this, the process is:

# Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all are, so they're unlikely to contain any useful data)

# Resubmit any container requests for which you want the output collections regenerated 

There's tool script that can be used to generate a report to help with this task in the Arvados repository at "arvados/tools/keep-xref/keep-xref.py":https://github.com/curoverse/arvados/blob/master/tools/keep-xref/keep-xref.py