Feature #14406
closed
[SDK] Go collection uses copy remote block to local keepstore
Description
Go SDK Collection uses "refresh keep signature" API (#14199) to convert +R token signatures into +A signatures.
Only supports copying block from remote clusters to home cluster. Only supports creating collections on the local cluster.
The Go keep client needs a new method to access the "refresh keep signature" API.
The Collection will check if there are any locators with remote signatures (+R) and uses the refresh keep signature API before creating / updating a collection on the home cluster.
Make sure that crunch-run uses this when copying inputs to outputs (eg when a collection is mounted within the output directory).
Updated by Peter Amstutz over 6 years ago
- Related to Feature #14259: [SDK] Python collection class uses copy remote block to local keepstore added
Updated by Tom Morris about 6 years ago
- Target version changed from To Be Groomed to 2018-11-14 Sprint
- Story points set to 2.0
Updated by Tom Clegg about 6 years ago
- Related to Feature #14199: [keepstore] copy block from remote keepstore to local keepstore added
Updated by Tom Clegg about 6 years ago
- Category set to Keep
- Status changed from New to In Progress
14406-remote-data-in-output @ 4ed6626cfba06e924f77dc54e1fe704eff1d113c
Updated by Lucas Di Pentima about 6 years ago
I believe there's an easy optimization that can be used for those collections with many files per block, LocalLocator could somehow cache previous copied remote blocks to avoid making duplicate HEAD requests, what do you think?
Updated by Tom Clegg about 6 years ago
Good point.
Caching in LocalLocator itself has more potential, but might be hard to get right (the caller might change the client's token between calls, the cache might need to be purged in case the client is long-lived, etc). So I did it in sync() to take care of the most common case, where a block is referenced by many files and the files are all in the same directory.
14406-remote-data-in-output @ b8035155b7db4b7ed11babde71e7d82c3d09fa6d
Updated by Tom Clegg about 6 years ago
- Status changed from In Progress to Resolved