Bug #7161

[SDKs] PySDK supports any Keep service type, using proxy replication logic for non-disk types

Added by Brett Smith over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
SDKs
Target version:
Start date:
09/23/2015
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

Functional requirements:

  • The Python SDK Keep client will talk to all accessible Keep services, regardless of service type.
  • The client will send the X-Keep-Desired-Replication header in all PUT requests, and respect the X-Keep-Replicas-Stored header in the response, regardless of service type (currently sends the request header only to proxies).
  • The client only has one thread at a time writing to non-disk services: If there are any non-"disk" services in the "accessible" list, use ThreadLimiter(1).

Future work: Anticipate API server telling us max replication for each service, meanwhile say 1 for disks and 3 for others. Let a thread start writing only when sum(max replication) < copies_todo, i.e., we will need to write more copies even if all currently writing threads succeed with max replication.

TBD: How should KeepClient choose a timeout for non-disk, non-proxy services?


Subtasks

Task #7377: Review branch 7161-support-all-service-typesResolvedRadhika Chippada


Related issues

Related to Arvados - Feature #7159: [Keep] Implement an Azure blob storage volume in keepstoreResolved08/28/2015

Related to Arvados - Bug #7180: [Keep] [SDKs] Logic about which services to try should not reference service_typeClosed09/01/2015

Associated revisions

Revision b4112f2e
Added by Radhika Chippada over 5 years ago

closes #7161
Merge branch '7161-support-all-service-types'

History

#1 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)

#2 Updated by Tom Clegg over 5 years ago

  • Description updated (diff)
  • Story points set to 1.0

#3 Updated by Brett Smith over 5 years ago

  • Target version changed from Arvados Future Sprints to 2015-09-16 sprint

#4 Updated by Brett Smith over 5 years ago

  • Target version changed from 2015-09-16 sprint to 2015-09-30 sprint

#5 Updated by Brett Smith over 5 years ago

  • Assigned To set to Radhika Chippada

#6 Updated by Brett Smith over 5 years ago

See #7235 for more thoughts about the timeout issue.

#7 Updated by Brett Smith over 5 years ago

  • Target version changed from 2015-09-30 sprint to Arvados Future Sprints

#8 Updated by Radhika Chippada over 5 years ago

  • Status changed from New to In Progress

#9 Updated by Radhika Chippada over 5 years ago

  • Target version changed from Arvados Future Sprints to 2015-09-30 sprint

#10 Updated by Tom Clegg over 5 years ago

The attribute "thread_count" seems to be:
  • None if we should use one thread per copy wanted
  • 1 if we should use 1 thread

Perhaps it would be easier to follow if we called it "maxReplicasPerService" and set it to 1 or None (None signifying "unknown")?

How about

-(True != ks.get('read_only'))
+not ks.get('read_only')

#11 Updated by Radhika Chippada over 5 years ago

Addressed all those comments. Thanks.

#12 Updated by Tom Clegg over 5 years ago

ee9c164 LGTM

(Except the commit message "working" isn't very helpful. Maybe this branch should rebase & squash into one commit?)

#13 Updated by Radhika Chippada over 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:b4112f2e4220312c71c5c3d288f888133a5ba41a.

Also available in: Atom PDF