Bug #6358
closed
[SDKs] Python Keep client uses wrong probe order for put()
Added by Peter Amstutz almost 9 years ago.
Updated over 8 years ago.
Description
I am running tests that writes a bunch of miscellaneous blocks to Keep. I have three keep services. PUT and GET are supposed to use the same algorithm to generate the sort order used to select which keep servers to store which blocks. This means that if I PUT a set of blocks and there are no errors or changes to the keep service configuration, every GET on that same set of blocks should go to the correct server the first time; there should be no 404s.
However I am seeing 404s on one of the keepstore services in my local install using the Python SDK (it does succeed in finding the block on another keepstore), which means the it is either sometimes putting the block on the wrong keep service, or it is trying the keep services in the wrong order.
- Subject changed from [SDK] Not always trying the correct server first to [SDK] Not always trying the correct keep server first
- Description updated (diff)
On further investigation, I don't think weighted_service_roots is at fault, I think there might be a race condition:
One possibility is that it starts writing to server A and server B, server A is a little slow to respond but server B accepts the block, so it goes on and writes to server C while still waiting for server A. However, if that were happening, the write to server A could still go through resulting in 3 replicas, not 2.
It could also be that it is actually the thread for writing to server A that is slow to start, so the threads writing to server B and server C complete before the thread for writing to server A get going?
Running arv-put and arv-get with ARVADOS_DEBUG=2 should generate enough log messages to see whether the servers are attempted in the right order, and which of arv-put, arv-get, and keepstore are doing something wrong (or slow).
When probe order is A,B,C and A is slow to respond, the correct behavior is to store on B and C as quickly as possible and move on. So it's possible this is not a client bug.
Does this happen with larger blocks, or only with small blocks? For small blocks, lookup+connection-setup time can be comparable to transfer+touch/store time, making out-of-order races more likely.
- Status changed from New to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:4f77c7788c6fc3a6cc9cd90ff231d837fdec7cc4.
- Status changed from Resolved to New
- Project changed from 35 to Arvados
- Subject changed from [SDK] Not always trying the correct keep server first to [SDKs] Put clients giving up on Keep servers quickly enough to thwart the probe list
- Category set to SDKs
- Target version changed from Bug Triage to Arvados Future Sprints
Updating the subject to reflect what sounds like the more likely issue here. I'm doing that understanding, like Tom said, this might not be a client bug at all. The clients might be doing the right thing, and the fix to the bad block distribution might be "Write Data Manager."
- Subject changed from [SDKs] Put clients giving up on Keep servers quickly enough to thwart the probe list to [SDKs] Python Keep client uses wrong probe order for put()
Reverting subject after confirming there is a real probe order bug (by writing a failing test and a fix).
(As it turns out, we were testing that the "last error from each server" list was being reported in the correct probe order, but we weren't testing that that we were encountering the errors in the correct probe order.)
6358-put-rendezvous
- Fix bug, test exact ordering for single-thread case: 25af384
- Test partial ordering for multiple-thread case: f269149
- Status changed from New to In Progress
6358-put-rendezvou:
As a matter of opinion, this seems like a missed opportunity to redesign multi threaded writes in keep client, e.g. a work queue design could would be simpler and avoid the need for racing writer threads to coordinate their execution order in the first place.
However, as a fix for the immediate problem at hand, it looks good to me.
Peter Amstutz wrote:
As a matter of opinion, this seems like a missed opportunity to redesign multi threaded writes in keep client, e.g. a work queue design could would be simpler and avoid the need for racing writer threads to coordinate their execution order in the first place.
However, as a fix for the immediate problem at hand, it looks good to me.
I agree there are refactoring opportunities here, but yes, a bugfix seemed more important. Hopefully these new tests will make it easier/safer to refactor when we get to it.
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:d379c467be58c66b2f1e7acafc97634b269a1542.
- Target version deleted (
Arvados Future Sprints)
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by