Arvados

background¶

The most obvious benefit of this change is that, even when keepstore is busy, the process of probing N volumes to find an existing block can proceed quickly, using the OS's filesystem cache, instead of waiting in every volume's serialize queue along the way. With the current implementation, a keepstore can easily accumulate a queue of PUT requests (each consuming a decent chunk of RAM) that are just waiting to find out whether the blocks already exist -- and of course they'll wait in the queue an N+1st time when it's time to actually write them to disk.

solution / implementation¶

Suggest refactoring the "serialize" implementation to use a sync.Mutex instead of passing channels to a goroutine over a "queue" channel.

In Read, before calling ioutil.ReadFile or doing any locking, do a stat() to see whether the file exists. If not, return an error without acquiring the volume lock.

In Read and Write, just before reading or writing data:

if v.mutex {
  v.mutex.Lock()
  defer v.mutex.Unlock()
}

Then we can rename Write to Put, rename Read to Get, and delete the old Get and Put wrappers, IOHandler, IORequest, IOResponse, and KeepGet/KeepPut constants.