Idea #8937
Updated by Brett Smith about 8 years ago
arv-put does not check the validity of the permission signatures on its blocks when it resumes from cache. When it resumes from a cache with invalid permission signatures, it will continue uploading blocks, but the ultimate collection creation at the end of the process will fail. Update the Python SDK: * Add a head() method to the KeepClient class. It takes a block locator and num_retries as arguments. an argument. It sends a HEAD request for that block to all accessible Keep services in the usual rendezvous order. It returns True if one service returns 200 OK, and raises an exception if it never gets that response. It retries following the same logic as the get() method. (what?). * Add tests for this method that check behavior when: ** One accessible service returns 200 OK. ** The last accessible service returns 200 OK. ** All accessible services return a permanent error. ** All accessible services return an error. Some of those errors are temporary. On a second request, at least one of the services returns 200 OK. ** All accessible services return temporary errors, enough to exhaust the number of retries. Update arv-put: * If the cache is otherwise usable (the file list is the same, the files are unchanged, etc.), use KeepClient's new head method to check the first block locator in the manifest in the cache. ** The first will always be the oldest and most likely to fail, so it is the best one to check. We talked about potentially checking all of them, and that is more thorough, but it's also potentially much more expensive. ** The rest of the cache checks are much cheaper than the HEAD request. We want to do this last, because if the cache is invalid for any other reason, we can notice and restart much faster. * If the head request confirms the block locator is still valid, continue from the cache as before. * Otherwise, invalidate the cache and start from cache. * Test that arv-put behaves correctly: ** In existing tests that arv-put resumes from a valid cache, update those tests to simulate a 200 OK response to the HEAD request. The rest of the tested behavior should be preserved as before. ** Write a test to verify that arv-put invalidates the cache in a situation where the HEAD request fails.