Arvados

Background:

Currently sdk/go/keepclient polls the /arvados/v1/keep_services/accessible endpoint (which controller proxies through to railsapi) to find keepstore endpoints.

When the client is configured using environment variables, ARVADOS_KEEP_SERVICES can be used to override discovery. However, when the client is configured using /etc/arvados/config.yml, ARVADOS_KEEP_SERVICES is ignored.

This makes it unnecessarily awkward to override only (for example) keep-web's upstream keepstore services for testing/troubleshooting purposes: in order for keep-web to use alternate backends, one would need to also run a controller+railsapi pair using the altered config.

Proposed change:

System services that act as keep clients (i.e., Go programs that use arvados.NewClientFromConfig()) should obey the ARVADOS_KEEP_SERVICES environment variable if set, otherwise use Services.Keepstore.InternalURLs from the config file.

They could also periodically (once an hour?) check the discovery endpoint and log a warning if it doesn't match. OTOH this might be redundant with the existing config-sync approach (system services report in-use config file checksums via metrics, and healthcheck reports if they aren't all equal).