Arvados

Recovering lost data » History » Revision 5

Revision 4 (Tom Clegg, 04/26/2019 08:08 PM) → Revision 5/7 (Tom Morris, 05/08/2019 06:30 PM)

h1. Recovering Lost Data 

 h2. Untrashing lost blocks 

 In some cases it is possible to recover data blocks that have been trashed by keep-balance (due to a bug like #15148, or an install/config error). 

 If you suspect blocks have been trashed erroneously, you should immediately: 
 # On all keepstore servers: set EmptyTrashInterval to a long time like 2400h 
 # On all keepstore servers: restart keepstore 
 # Stop the keep-balance service 

 When you think you have corrected the underlying problem, you should: 
 # Set LostBlocksFile to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt") in your keep-balance config 
 # Start keep-balance 

 After keep-balance completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this: 

 <pre><code class="bash"> 
 #!/bin/bash 
 set -e 

 # see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml 
 token=xxxxxxx-your-system-auth-token-xxxxxxx 

 # all keep server hostnames 
 hosts=(keep0 keep1 keep2 keep3 keep4 keep5) 

 while read hash pdhs; do 
     echo "${hash}" 
     for h in ${hosts[@]}; do 
         if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then 
             echo "${hash} ok ${host}" 
         fi 
     done 
 done < /tmp/keep-balance-lost-blocks.txt 
 </code></pre> 

 Obviously this could be improved upon with increased parallelism for large scale tasks, if needed. Any blocks which were successfully untrashed can be removed from the list of blocks and collections which need to be recovered. 

 h2. Regenerating missing blocks 

 For blocks which were trashed long enough that they've been deleted, it's possible to regenerate them by rerunning the workflows which generated them. To do this, the process is: 
 # Delete the affected collections so that job reuse doesn't attempt to reuse them (it's likely that if one block is missing, they all our, so they're unlikely to contain any useful data) 
 # Resubmit any container requests that you want to regenerate the output collections for  

 There's tool script that can be used to generate a report to help with this task in the Arvados repository at arvados/tools/keep-xref/keep-xref.py