Project

General

Profile

Bug #6221

Updated by Tom Clegg over 8 years ago

By design, Data Manager deletes unneeded blocks by sending "trash lists" to the keepstore servers. 

 Currently, Data Manager computes lists of trashable blocks, but doesn't send them to the keepstore servers. 

 Merge !#1 should include: Before this can be implemented and deployed: 
 * trash_worker.go must respect the @never_delete@ flag, and this must be tested. When this is set, it should log a message (including the locator of the block being deleted) instead of deleting the file. 
 * *Temporarily* -- until datamanager has an integration test -- modify keepstore so (except for the test suite) it is impossible to make @never_delete==false@. 
 * data manager sends must send its trash lists to keep servers 
 ** the current code identifies only a subset of trash: blocks that are not referenced by any collection. It doesn't identify overreplicated blocks as trash. This is fine for now -- we're just concentrating on getting the list it _does_ generate to propagate to the keepstore servers. 

 Merge !#2 should include: 
 * data manager must have _at least_ one comprehensive integration test covering a miniature version of normal operation: 
 ** bring up api, keepstore, and keepproxy services 
 ** store some collections 
 ** run data manager 
 ** wait for all keepstores to finish working their trash and pull lists 
 ** delete some of the collections 
 ** reduce replication on some of the collections 
 ** run data manager again 
 ** bring up at least one more keepstore 
 ** turn off one keepstore 
 ** run data manager again 
 ** wait for all keepstores to finish working their trash and pull lists 
 ** make sure all non-garbage blocks are stored at the correct replication level, and in their optimal positions 

Back