Project

General

Profile

Actions

Bug #5745

closed

[Keep] keepstore should serialize file reads/writes, but not directory lookups, when -serialize=true

Added by Tom Clegg over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
0.5

Description

background

The most obvious benefit of this change is that, even when keepstore is busy, the process of probing N volumes to find an existing block can proceed quickly, using the OS's filesystem cache, instead of waiting in every volume's serialize queue along the way. With the current implementation, a keepstore can easily accumulate a queue of PUT requests (each consuming a decent chunk of RAM) that are just waiting to find out whether the blocks already exist -- and of course they'll wait in the queue an N+1st time when it's time to actually write them to disk.

solution / implementation

Suggest refactoring the "serialize" implementation to use a sync.Mutex instead of passing channels to a goroutine over a "queue" channel.

In Read, before calling ioutil.ReadFile or doing any locking, do a stat() to see whether the file exists. If not, return an error without acquiring the volume lock.

In Read and Write, just before reading or writing data:

if v.mutex {
  v.mutex.Lock()
  defer v.mutex.Unlock()
}

Then we can rename Write to Put, rename Read to Get, and delete the old Get and Put wrappers, IOHandler, IORequest, IOResponse, and KeepGet/KeepPut constants.


Subtasks 1 (0 open1 closed)

Task #5924: review 5745-serialize-content-onlyResolvedTom Clegg05/07/2015Actions
Actions

Also available in: Atom PDF