Project

General

Profile

Actions

Bug #5745

closed

[Keep] keepstore should serialize file reads/writes, but not directory lookups, when -serialize=true

Added by Tom Clegg almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Story points:
0.5

Description

background

The most obvious benefit of this change is that, even when keepstore is busy, the process of probing N volumes to find an existing block can proceed quickly, using the OS's filesystem cache, instead of waiting in every volume's serialize queue along the way. With the current implementation, a keepstore can easily accumulate a queue of PUT requests (each consuming a decent chunk of RAM) that are just waiting to find out whether the blocks already exist -- and of course they'll wait in the queue an N+1st time when it's time to actually write them to disk.

solution / implementation

Suggest refactoring the "serialize" implementation to use a sync.Mutex instead of passing channels to a goroutine over a "queue" channel.

In Read, before calling ioutil.ReadFile or doing any locking, do a stat() to see whether the file exists. If not, return an error without acquiring the volume lock.

In Read and Write, just before reading or writing data:

if v.mutex {
  v.mutex.Lock()
  defer v.mutex.Unlock()
}

Then we can rename Write to Put, rename Read to Get, and delete the old Get and Put wrappers, IOHandler, IORequest, IOResponse, and KeepGet/KeepPut constants.


Subtasks 1 (0 open1 closed)

Task #5924: review 5745-serialize-content-onlyResolvedTom Clegg05/07/2015Actions
Actions #1

Updated by Tom Clegg almost 9 years ago

  • Description updated (diff)
  • Category set to Keep
Actions #2

Updated by Tom Clegg almost 9 years ago

  • Target version changed from Arvados Future Sprints to 2015-05-20 sprint
Actions #3

Updated by Tom Clegg almost 9 years ago

  • Status changed from New to In Progress
Actions #4

Updated by Peter Amstutz almost 9 years ago

867263b 5745-serialize-content-only LGTM

Actions #5

Updated by Peter Amstutz almost 9 years ago

  • Assigned To set to Tom Clegg
Actions #6

Updated by Tom Clegg almost 9 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:fde21d34d011af2123668983c559632221390fd4.

Actions

Also available in: Atom PDF