Feature #14406

[SDK] Go collection uses copy remote block to local keepstore

Added by Peter Amstutz 11 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
11/05/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0
Release:
Release relationship:
Auto

Description

Go SDK Collection uses "refresh keep signature" API (#14199) to convert +R token signatures into +A signatures.

Only supports copying block from remote clusters to home cluster. Only supports creating collections on the local cluster.

The Go keep client needs a new method to access the "refresh keep signature" API.

The Collection will check if there are any locators with remote signatures (+R) and uses the refresh keep signature API before creating / updating a collection on the home cluster.

Make sure that crunch-run uses this when copying inputs to outputs (eg when a collection is mounted within the output directory).


Subtasks

Task #14443: Review 14406-remote-data-in-outputResolvedLucas Di Pentima


Related issues

Related to Arvados - Feature #14259: [SDK] Python collection class uses copy remote block to local keepstoreResolved10/30/2018

Related to Arvados - Feature #14199: [keepstore] copy block from remote keepstore to local keepstoreResolved10/04/2018

Associated revisions

Revision c72b6bb7
Added by Tom Clegg 11 months ago

Merge branch '14406-remote-data-in-output'

refs #14406

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 11 months ago

  • Related to Feature #14259: [SDK] Python collection class uses copy remote block to local keepstore added

#2 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#3 Updated by Tom Morris 11 months ago

  • Target version changed from To Be Groomed to 2018-11-14 Sprint
  • Story points set to 2.0

#4 Updated by Tom Clegg 11 months ago

  • Assigned To set to Tom Clegg

#5 Updated by Tom Clegg 11 months ago

  • Related to Feature #14199: [keepstore] copy block from remote keepstore to local keepstore added

#6 Updated by Tom Clegg 11 months ago

  • Category set to Keep
  • Status changed from New to In Progress

14406-remote-data-in-output @ 4ed6626cfba06e924f77dc54e1fe704eff1d113c

#7 Updated by Lucas Di Pentima 11 months ago

I believe there's an easy optimization that can be used for those collections with many files per block, LocalLocator could somehow cache previous copied remote blocks to avoid making duplicate HEAD requests, what do you think?

#8 Updated by Tom Clegg 11 months ago

Good point.

Caching in LocalLocator itself has more potential, but might be hard to get right (the caller might change the client's token between calls, the cache might need to be purged in case the client is long-lived, etc). So I did it in sync() to take care of the most common case, where a block is referenced by many files and the files are all in the same directory.

14406-remote-data-in-output @ b8035155b7db4b7ed11babde71e7d82c3d09fa6d

#9 Updated by Lucas Di Pentima 11 months ago

This LGTM, thanks!

#10 Updated by Tom Clegg 11 months ago

  • Status changed from In Progress to Resolved

#11 Updated by Tom Morris 10 months ago

  • Release set to 14

Also available in: Atom PDF