[SDK] Python client tries to cache discovery document at the same local path regardless of user
The Python SDK instructs httplib2 to cache the discovery document in
~/.cache/arvados/discovery. However, the google API client code also has its own cache logic, which caches the discovery document in
This is a problem particularly on a multi-user system such as a shell node. The file gets created in a global location like /tmp, then only the original user can update the file.
From looking at the google-api-python-client code, the destination is chosen as
os.path.join(tempfile.gettempdir(), FILENAME) where
FILENAME = 'google-api-python-client-discovery-doc.cache'
#6 Updated by Peter Amstutz almost 3 years ago
I think we can use the
cache_discovery option to build() to disable the Google API client caching.
#8 Updated by Peter Amstutz almost 3 years ago
Also want to do one of:
a) continue to use httplib2 caching, which is apparently not multithread/multiprocess safe
b) provide our own httplib2 cache which is multithread/multiprocess safe
c) provide a google api client cache object (with
build(cache=XXX)) which is multithread/multiprocess safe:
#15 Updated by Peter Amstutz almost 3 years ago
Tom Morris wrote:
I take it this got instagroomed while I was gone.
Does this mean that #10669 is fixed (and can be closed) as well?
No, this only disables google-api-client caching so that it uses the httplib2 cache as originally intended. The bug in #10669 is that the httplib2 cache is not multi-thread/multi-process safe, and which makes it possible to get corrupted. The google-api-client cache actually does do file locking, but puts the cache file in the global
/tmp and doesn't provide a way to change that except for changing the global tempfile path (which has side effects for everything using the tempfile module).