Feature #13993

[API] Fetch remote-hosted collection by UUID

Added by Peter Amstutz 12 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
09/07/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0
Release:
Release relationship:
Auto

Description

Enable a client to retrieve a collection record (with a given UUID) from its home cluster, even if the collection is hosted on a different cluster in a federation.

This issue covers the arvados-controller changes required for basic functionality.
  • proxy "get collection" requests with foreign UUIDs to the appropriate remote cluster (similar to #13493)
  • alter the signature hints in the returned record so keepstore will be able to get the data from the remote cluster (see Federated collections wiki)
This issue does not include
  • cache optimizations
  • changes to keepstore (#13994)

Subtasks

Task #14053: Review 13993-federated-collectionResolvedPeter Amstutz


Related issues

Related to Arvados - Feature #13493: Federated record retrievalResolved06/28/2018

Related to Arvados - Feature #13994: [Keepstore] Fetch blocks from federated clustersResolved09/12/2018

Related to Arvados - Feature #14087: [controller] Fetch remote-hosted collections by PDHResolved09/10/2018

Associated revisions

Revision 2b0b0657
Added by Peter Amstutz 10 months ago

Merge branch '13993-federated-collection' refs #13993

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 12 months ago

  • Subject changed from [API] Fetch remote-hosted container by UUID to [API] Fetch remote-hosted collection by UUID
  • Description updated (diff)

#2 Updated by Peter Amstutz 12 months ago

#3 Updated by Peter Amstutz 12 months ago

  • Related to Feature #13994: [Keepstore] Fetch blocks from federated clusters added

#4 Updated by Peter Amstutz 12 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 12 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 12 months ago

  • Description updated (diff)

#7 Updated by Tom Morris 12 months ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 2.0

#8 Updated by Tom Clegg 12 months ago

  • Description updated (diff)

#9 Updated by Tom Morris 11 months ago

  • Target version changed from Arvados Future Sprints to 2018-09-05 Sprint

#10 Updated by Tom Morris 11 months ago

  • Assigned To set to Fuad Muhic

#11 Updated by Peter Amstutz 11 months ago

  • Assigned To deleted (Fuad Muhic)

#13 Updated by Peter Amstutz 11 months ago

  • Related to Feature #14087: [controller] Fetch remote-hosted collections by PDH added

#14 Updated by Peter Amstutz 11 months ago

  • Target version changed from 2018-09-05 Sprint to 2018-09-19 Sprint

#15 Updated by Peter Amstutz 11 months ago

  • Subject changed from [API] Fetch remote-hosted collection by UUID to [API] Fetch remote-hosted collection by UUID
  • Assigned To set to Peter Amstutz

#17 Updated by Lucas Di Pentima 11 months ago

Some comments:

  • On file lib/controller/federation.go
  • I think it would be convenient to update the locator format documentation to describe a +R hint
  • We have a previous signed locator Regex on sdk/go/keepclient/perms.go:68 that looks more correct (because I believe the current one doesn’t allow +A hints that aren’t signature hints), should we export it so it can be reused here?
  • Tests are failing

#18 Updated by Peter Amstutz 11 months ago

Lucas Di Pentima wrote:

Some comments:

  • On file lib/controller/federation.go
    • Line 96 - Small detail: I think just multiplying by 1.1 would more readable

I was trying to avoid converting to double precision and back but I agree that's a silly micro-optimization in this case at the expense of readability

See below

  • I think it would be convenient to update the locator format documentation to describe a +R hint

I have a documentation story to write up federation features (the task on #13493 assigned to me), maybe I can do it there?

  • We have a previous signed locator Regex on sdk/go/keepclient/perms.go:68 that looks more correct (because I believe the current one doesn’t allow +A hints that aren’t signature hints), should we export it so it can be reused here?

I consolidated the regex so the one in keepclient can be used by federation.go

  • Tests are failing

Works for me... Kicked off a run here: https://ci.curoverse.com/view/Developer/job/developer-run-tests/881/

#20 Updated by Lucas Di Pentima 10 months ago

Just one question: Would it be convenient to also add a test for a local collection request, so we test both sides?
Apart from that, it LGTM.

#21 Updated by Peter Amstutz 10 months ago

Lucas Di Pentima wrote:

Just one question: Would it be convenient to also add a test for a local collection request, so we test both sides?
Apart from that, it LGTM.

Added TestGetLocalCollection @ 9d2e1158e961801db714236c6942bd3596b867e7

#22 Updated by Lucas Di Pentima 10 months ago

Thanks! This lgtm.

#23 Updated by Peter Amstutz 10 months ago

  • Status changed from New to Resolved

#24 Updated by Tom Morris 8 months ago

  • Release set to 14

Also available in: Atom PDF