Project

General

Profile

Actions
Copy link

Bug #7161

closed

[SDKs] PySDK supports any Keep service type, using proxy replication logic for non-disk types

Added by Brett Smith over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
SDKs
Target version:
2015-09-30 sprint
Story points:
1.0

Description

Functional requirements:

  • The Python SDK Keep client will talk to all accessible Keep services, regardless of service type.
  • The client will send the X-Keep-Desired-Replication header in all PUT requests, and respect the X-Keep-Replicas-Stored header in the response, regardless of service type (currently sends the request header only to proxies).
  • The client only has one thread at a time writing to non-disk services: If there are any non-"disk" services in the "accessible" list, use ThreadLimiter(1).

Future work: Anticipate API server telling us max replication for each service, meanwhile say 1 for disks and 3 for others. Let a thread start writing only when sum(max replication) < copies_todo, i.e., we will need to write more copies even if all currently writing threads succeed with max replication.

TBD: How should KeepClient choose a timeout for non-disk, non-proxy services?

Subtasks 1 (0 open — 1 closed)

Task #7377: Review branch 7161-support-all-service-typesResolvedRadhika Chippada09/23/2015Actions

Related issues

Related to Arvados - Feature #7159: [Keep] Implement an Azure blob storage volume in keepstoreResolvedTom Clegg08/28/2015Actions
Related to Arvados - Bug #7180: [Keep] [SDKs] Logic about which services to try should not reference service_typeClosed09/01/2015Actions
Actions
Copy link
 #1

Updated by Tom Clegg over 8 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Tom Clegg over 8 years ago

  • Description updated (diff)
  • Story points set to 1.0
Actions
Copy link
 #3

Updated by Brett Smith over 8 years ago

  • Target version changed from Arvados Future Sprints to 2015-09-16 sprint
Actions
Copy link
 #4

Updated by Brett Smith over 8 years ago

  • Target version changed from 2015-09-16 sprint to 2015-09-30 sprint
Actions
Copy link
 #5

Updated by Brett Smith over 8 years ago

  • Assigned To set to Radhika Chippada
Actions
Copy link
 #6

Updated by Brett Smith over 8 years ago

See #7235 for more thoughts about the timeout issue.

Actions
Copy link
 #7

Updated by Brett Smith over 8 years ago

  • Target version changed from 2015-09-30 sprint to Arvados Future Sprints
Actions
Copy link
 #8

Updated by Radhika Chippada over 8 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #9

Updated by Radhika Chippada over 8 years ago

  • Target version changed from Arvados Future Sprints to 2015-09-30 sprint
Actions
Copy link
 #10

Updated by Tom Clegg over 8 years ago

The attribute "thread_count" seems to be:
  • None if we should use one thread per copy wanted
  • 1 if we should use 1 thread

Perhaps it would be easier to follow if we called it "maxReplicasPerService" and set it to 1 or None (None signifying "unknown")?

How about 

-(True != ks.get('read_only'))
+not ks.get('read_only')

Actions
Copy link
 #11

Updated by Radhika Chippada over 8 years ago

Addressed all those comments. Thanks.

Actions
Copy link
 #12

Updated by Tom Clegg over 8 years ago

ee9c164 LGTM

(Except the commit message "working" isn't very helpful. Maybe this branch should rebase & squash into one commit?)

Actions
Copy link
 #13

Updated by Radhika Chippada over 8 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:b4112f2e4220312c71c5c3d288f888133a5ba41a.

Actions
Copy link

Also available in: Atom PDF