Bug #19277
closed
Arvados client inside container should use local keepstore
Added by Tom Clegg over 2 years ago.
Updated over 2 years ago.
Release relationship:
Auto
Description
Currently, when crunch-run starts a local keepstore process, it gets used by arv-mount, but is not advertised to, or usable from, processes inside the container.
The result is that, when many containers are running, a container that uses the Arvados API natively (as lightning does) oversaturates the fixed keepstore gateway nodes while the closer/faster keepstore processes sit idle. (Example: #19236#note-9)
Implementation
- local keepstore listens on non-loopback interface(s) (currently listens only on localhost, which is inaccessible from the container)
- crunch-run passes suitable ARVADOS_KEEP_SERVICES env var into the container
Choosing the best listening address is a bit awkward.
In the docker case, the docker0 interface seems ideal: it's known before the container starts, the container can connect to it, but it's not routable from outside the worker host.
In the singularity case, the host side of the host/container private network link doesn't exist until we ask singularity to start a container, which means we've already told singularity what we want the ARVADOS_KEEP_SERVICES env var to be.
I think the following approach should work consistently for both cases:
- get local IP addresses from
/proc/$$/net/fib_trie (we have a function for this in lib/crunchrun/singularity.go)
- sort numerically (10.2.2.2 < 10.10.10.10)
- choose the first address that is not loopback (127/8), VPN (100.64/10), or link-local (169.254/16)
- use this (instead of localhost) as the listening address for keepstore, and the KEEP_SERVICES url passed to arv-mount and into the container
Looks good, please merge. Thanks!
- Status changed from In Progress to Resolved
I tried to cherry-pick this on to 2.4-staging and it didn't apply cleanly, so I'm rejecting it for 2.4.3
cherry-picked to 2.4-release as 6e992b73bf60a23b2ca10ca9694e5dff4d1497cc
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by