Idea #9551
closed
[SDKs] Python SDK supports an ARVADOS_KEEP_SERVICES environment variable
Description
ARVADOS_KEEP_SERVICES is an environment variable that lets the client set its own list of Keep services, instead of fetching it from the API server. It is a space-separated list of Keep API endpoint URLs. Those endpoints are expected to be non-disk services: the client should assume it does not know how many replicas the service will store, should send the X-Keep-Desired-Replicas request header, and respect the X-Keep-Replicas-Stored response header.
A client should pay attention to this env var if its api_host/token/insecure settings were also taken from the environment. IOW, treat it as an additional client config entry, rather than checking the environment variable from the depths of KeepClient.
Functional requirements:
- When the variable is set, it does not query the API server for the Keep services list, or respect the ARVADOS_KEEP_PROXY environment variable. Instead it uses the value of the variable as the services list, following the rules above.
- When the variable is unset or empty, the Keep client should behave as it currently does: if ARVADOS_KEEP_PROXY is set, it treats that as a single service to use. Otherwise, it gets the list of Keep services from the API server.
Related issues
Updated by Brett Smith over 8 years ago
- Target version set to Arvados Future Sprints
Updated by Tom Clegg over 8 years ago
- Assigned To set to Tom Clegg
- Target version changed from Arvados Future Sprints to 2016-08-03 sprint
Updated by Lucas Di Pentima over 8 years ago
- Assigned To changed from Tom Clegg to Lucas Di Pentima
Updated by Lucas Di Pentima over 8 years ago
- Status changed from New to In Progress
Updated by Lucas Di Pentima over 8 years ago
2aefffe61af4bb93659ca1cd9a9a027858658bec
Added support of ARVADOS_KEEP_SERVICES env var by enhancing the code that takes care of setting up a proxy keep service.
Updated by Tom Clegg over 8 years ago
- ARVADOS_KEEP_SERVICES
- "proxy" kwarg
- ARVADOS_KEEP_PROXY
The docstring claims that proxy='' prevents KeepClient from using ARVADOS_KEEP_PROXY, so we should probably make it override ARVADOS_KEEP_SERVICES too.
Should also update that docstring to mention the new env var.
Should update the use of ARVADOS_KEEP_PROXY in the test suite to use the new env var instead.
Nit: proxy_uris.index(uri) seems like an inefficient way to get the loop index. Also, if the list has two identical entries, we'll have two identical UUIDs, which could cause strangeness elsewhere. How about doing an idx++ each time through the loop? (Or some other more Pythonic equivalent of Go's "for i, x := range a"?)
Nit: In the test case, might as well use examples with realistic URI formats, like http://10.1.0.1 and https://bar.example.com:12345/
Updated by Lucas Di Pentima over 8 years ago
Tom Clegg wrote:
It looks like the new precedence order is
- ARVADOS_KEEP_SERVICES
- "proxy" kwarg
- ARVADOS_KEEP_PROXY
The docstring claims that proxy='' prevents KeepClient from using ARVADOS_KEEP_PROXY, so we should probably make it override ARVADOS_KEEP_SERVICES too.
Corrected. Now the proxy param has the priority again, and ARVADOS_KEEP_SERVICES is second.
Should also update that docstring to mention the new env var.
Updated.
Should update the use of ARVADOS_KEEP_PROXY in the test suite to use the new env var instead.
Done!
Nit: proxy_uris.index(uri) seems like an inefficient way to get the loop index. Also, if the list has two identical entries, we'll have two identical UUIDs, which could cause strangeness elsewhere. How about doing an idx++ each time through the loop? (Or some other more Pythonic equivalent of Go's "for i, x := range a"?)
Corrected. enumerate() is the function that I couldn't find before and is the mentioned Go equivalent.
Nit: In the test case, might as well use examples with realistic URI formats, like
http://10.1.0.1andhttps://bar.example.com:12345/
Corrected.
Also, added URL validation using urlparse module.
Thanks!
Updated by Lucas Di Pentima over 8 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:e31f6c9eb427302eb48150d431afcc962954b061.