Arvados

Idea #3705

Updated by Tom Clegg over 9 years ago

The lists of "blocks to delete" Both the trash collector and "blocks to pull from other keepstores" manager should be stored in implemented by a BlockWorkList type. Just like the existing PullList, common BlockManager package.    A BlockManager runs concurrently with Keepstore, and is responsible for maintaining a BlockWorkList has 
 * a current list 
 * a dedicated goroutine responsible of blocks that require outstanding attention (for trash collector, deleting; for setting and getting the current list pull manager, replicating). 

 Each item in the list has 
 * A @BlockWorker@ is a block hash 
 * some additional data (sending/accepting the right type here defined as @func(interface{})@ i.e. a function that takes a single generic object as its argument.    Note that a BlockWorker function is left up responsible for casting its argument to the callers) appropriate type for whatever action needs to be taken. 

 The BlockWorkList BlockManager interface provides these methods: 
 * @NewBlockWorkList()@ @Start(BlockWorker)@ 
 ** Returns Launches a concurrent listener that waits for a new block manager. Starts a new goroutine (which external callers don't interact with directly). list to be delivered via @SetList()@. The listener loops over the list and applies the @BlockWorker@ function to each block. 
 * @SetList()@ @SetList(container.List)@ 
 ** Replaces Delivers a new block list to the current listener. Upon receipt of the new list, the listener replaces any existing list and with the supplied list. (No effort is made to notify other threads that might be using old data. If they ask, we'll tell them.) new one, discarding any unprocessed requests, and begins processing it. 
 * @GetList()@ 
 ** Returns the listener's current block list. 
 * @Close()@ 
 ** Terminates Shuts down the goroutine. Subsequent calls listener. 

 Two different BlockManager objects are distinguished by: 
 * The @BlockWorker@ function that processes each item. 
 * The type of block action request that is stored in the block list passed to SetList and GetList() will return errors. SetList(). 

 Pseudocode for For example, to implement a trash collector: 
 <pre> 
 trashList := NewBlockWorkList() type DeleteRequest struct { 
     blockhash string 
     expires time.Time 
 trashList.SetList(...) } 

 go func() func DeleteBlock (item interface{}) { 
     for var r DeleteRequest 
     if r, ok := item.(DeleteRequest); ok { 
         latest := trashList.GetList() 
         (try to delete some blocks) vol.Delete(r.blockhash) 
     } 
 } 

 trash := block_manager.New() 
 trash.Start(DeleteBlock) 
 trash.SetList(blocks_to_delete) 
 </pre> 

 Future work: 
 * DeleteItem() method (a worker goroutine will want to do this when it decides that the work has been done). 
 * GetNonEmptyList(), if the current list is empty, block until it isn't. Then return GetList(). 

Back