Feature #14406
closed
[SDK] Go collection uses copy remote block to local keepstore
Added by Peter Amstutz over 6 years ago.
Updated about 6 years ago.
Release relationship:
Auto
Description
Go SDK Collection uses "refresh keep signature" API (#14199) to convert +R token signatures into +A signatures.
Only supports copying block from remote clusters to home cluster. Only supports creating collections on the local cluster.
The Go keep client needs a new method to access the "refresh keep signature" API.
The Collection will check if there are any locators with remote signatures (+R) and uses the refresh keep signature API before creating / updating a collection on the home cluster.
Make sure that crunch-run uses this when copying inputs to outputs (eg when a collection is mounted within the output directory).
- Related to Feature #14259: [SDK] Python collection class uses copy remote block to local keepstore added
- Description updated (diff)
- Target version changed from To Be Groomed to 2018-11-14 Sprint
- Story points set to 2.0
- Assigned To set to Tom Clegg
- Related to Feature #14199: [keepstore] copy block from remote keepstore to local keepstore added
- Category set to Keep
- Status changed from New to In Progress
I believe there's an easy optimization that can be used for those collections with many files per block, LocalLocator could somehow cache previous copied remote blocks to avoid making duplicate HEAD requests, what do you think?
Good point.
Caching in LocalLocator itself has more potential, but might be hard to get right (the caller might change the client's token between calls, the cache might need to be purged in case the client is long-lived, etc). So I did it in sync() to take care of the most common case, where a block is referenced by many files and the files are all in the same directory.
14406-remote-data-in-output @ b8035155b7db4b7ed11babde71e7d82c3d09fa6d
- Status changed from In Progress to Resolved
Also available in: Atom
PDF
Updated by 
Updated by Tom Morris 
Updated by 
Updated by