Arvados

Recovering lost data » History » Revision 3

Revision 2 (Nico César, 04/26/2019 06:32 PM) → Revision 3/7 (Tom Clegg, 04/26/2019 07:58 PM)

h1. Untrashing lost blocks 

 In some cases it is possible to recover data blocks that have been trashed by keep-balance (due to a bug or an install/config error). 

 If you suspect blocks have been trashed erroneously, you should immediately: 
 # On all keepstore servers: set EmptyTrashInterval to a long time like 2400h 
 # On all keepstore servers: restart keepstore 
 # Stop the keep-balance service 

 When you think you have corrected the underlying problem, you should: 
 # Set LostBlocksFile to a suitable value (perhaps "/tmp/keep-balance-lost-blocks.txt") in your keep-balance config 
 # Start keep-balance 

 After keep-balance completes its first sweep, inspect /tmp/keep-balance-lost-blocks.txt. If it's not empty, you can request all keepstores to untrash any blocks that are still recoverable with a script like this: 

 <pre><code class="bash"> 
 #!/bin/bash 
 set -e 

 # see Client.AuthToken in /etc/arvados/keep-balance/keep-balance.yml 
 token=xxxxxxx-your-system-auth-token-xxxxxxx 

 # all keep server hostnames 
 hosts=(keep0 keep1 keep2 keep3 keep4 keep5) 

 while read hash pdhs; hash; do 
     echo "${hash}" 
     for h in ${hosts[@]}; do 
         if curl -fgs -H "Authorization: Bearer $token" -X PUT "http://${h}:25107/untrash/$hash"; then 
             echo "${hash} ok ${host}" 
         fi 
     done 
 done < /tmp/keep-balance-lost-blocks.txt 
 </code></pre>