Arvados

Idea #3705

Updated by Tom Clegg over 9 years ago

The lists of "blocks to delete" and "blocks to pull from other keepstores" should be maintained stored in a BlockWorkList type. Just like the existing PullList, a BlockWorkList has 
 * a current list 
 * a dedicated goroutine responsible for setting and getting the current list 

 Each item in the list has 
 * a block hash 
 * some additional data (sending/accepting the right type here is left up to the callers) 

 The BlockWorkList interface provides these interfaces: methods: 
 * @NewBlockWorkList()@ 
 ** Returns a new BlockWorkList. block manager. Starts a new goroutine. goroutine (which external callers don't interact with directly). 
 * @ReplaceList@ @SetList()@ 
 ** Writing a list to this channel replaces Replaces the current list, if any, list with the supplied list. (The remainder of the old list (No effort is discarded. Any work still in progress on made to notify other threads that might be using old list items is unaffected.) data. If they ask, we'll tell them.) 
 * @NextItem@ @GetList()@ 
 ** Reading from this channel yields Returns the next item from the listener's current block list. (The returned item is also removed from the list.) 
 * @Close()@ 
 ** Terminates the goroutine, closes the channels. goroutine. Subsequent calls to SetList and GetList() will return errors. 

 Pseudocode for a trash collector: 
 <pre> 
 // Make a work list: 
 trashList := NewBlockWorkList() 

 // In an HTTP handler for "PUT /trash": 
 { 
     trashList.SetList(...) 
 } 

 // Start a worker: 
 go func(list BlockWorkList) func() { 
     for deleteRequest := range list.NextItem { 
         // Check timestamp and latest := trashList.GetList() 
         (try to delete the block. some blocks) 
     } 
 }(trashList) } 
 </pre> 

 Future work: 
 * DeleteItem() method (a worker goroutine will want to do this when it decides that the work has been done). 
 * GetNonEmptyList(), if the current list is empty, block until it isn't. Then return GetList(). 

Back