Arvados

A bit of history: In #2776 we started by having config knobs for setting a proxy host/port to advertise to external clients in place of real keep disks. Then we decided to "actually add the proxy disk to the database, and then use or exclude it for all results based on the configuration+header combination." We ended up using "set service_type to 'proxy'" as the configuration mechanism. Then the Python SDK started making various other decisions based on "service_type=='proxy'" (like timeouts, and whether to add an X-Keep-Desired-Replication header), and the Go SDK followed suit.

I think the keep_services APIs should give SDKs the answers to these questions:

• When I want to write a block, where should I send it? (If >1, use rendezvous.) (this is the "accessible" API, minus any readonly services)
• When I want to read a block that has no +K@{...} hint, where should I try to find it? (If >1, use rendezvous.) (this is the "accessible" API)
• When I want to read a block that has a +K@{uuid} hint, what scheme://host:port should I connect to? (this is the "index" API)
• When I need to write N more copies of a block, and I've already started sending it to service X, should I also start sending a copy to service Y? Or is service X capable of writing N copies itself? (we don't have this yet)

None of these should require any client-side logic based on whether service_type is disk, proxy, gateway, etc.

Appropriate timeouts are hard to predict in general. Putting this logic on the API side, and advertising suitable numbers with each keep service record, might be better than what we have now: you could configure longer timeouts at a site where the storage backend is far away, make them match your Nginx configuration, try tweaking timeouts without a deploy-everything-and-update-docker-images adventure, etc.