Arvados

Discovery doc retrieval is done by google-api-client. We ask it to use httplib2's built-in cache mechanism.

Unfortunately that built-in cache is "Not really safe to use if multiple threads or processes are going to be running on the same cache."

For example:

    def set(self, key, value):
        cacheFullPath = os.path.join(self.cache, self.safe(key))
        f = open(cacheFullPath, "wb")
        f.write(value)
        f.close()

https://github.com/jcgregorio/httplib2/blob/master/python3/httplib2/__init__.py#L666-L697

Fortunately it looks like we can make an object that supports the same interface as FileCache, and pass that to httplib2.Http() instead of "~/.cache/arvados/discovery".

10669-safe-http-cache @ 514cd364c3cb27b633c1368cd06d6a54927c98a8

• it ignores old corrupt files left behind by the default httplib2 implementation
• it can clean up its own mess by removing *.tmp with much less risk of deleting valuable stuff (even if it somehow ends up using a useful directory as a cache directory)

Did you try the multithreaded test_cache_crud against the regular httplib2 cache? Does the (unsafe) cache fail the test regularly, where the new cache passes?

For what its worth, from my cache directory:

Old filename:
qr1hi.arvadosapi.com,discovery,v1,apis,arvados,v1,rest,86e60ccc4da91c726aa80dfb73fa9cbf

New filename:
86e60ccc4da91c726aa80dfb73fa9cbf.tmp

Not really sure of the original purpose/intent of having the filename contain both a munged URL and the hash of the URL (except maybe to make debugging easier).

Content is the same except timestamps in the headers.

Ah yes, older versions of httplib2 used just the md5 but current versions use a munged url plus md5. And yeah, I assume the reason is for debugging, so you don't have to grep for ^content-location:.

Peter Amstutz wrote:

Did you try the multithreaded test_cache_crud against the regular httplib2 cache? Does the (unsafe) cache fail the test regularly, where the new cache passes?

(I hadn't before, but) I tried it, and yes, it fails horribly. Fixed up the test case so it prints a bunch of errors and then fails an assertion, instead of dumping a lot of stack traces.

10669-safe-http-cache @ c56743e301b49163a56482c13e49a01c9a0fd7dc