Feature #13993
closed
[API] Fetch remote-hosted collection by UUID
Description
Enable a client to retrieve a collection record (with a given UUID) from its home cluster, even if the collection is hosted on a different cluster in a federation.
This issue covers the arvados-controller changes required for basic functionality.- proxy "get collection" requests with foreign UUIDs to the appropriate remote cluster (similar to #13493)
- alter the signature hints in the returned record so keepstore will be able to get the data from the remote cluster (see Federated collections wiki)
- cache optimizations
- changes to keepstore (#13994)
Related issues
Updated by Peter Amstutz over 5 years ago
- Subject changed from [API] Fetch remote-hosted container by UUID to [API] Fetch remote-hosted collection by UUID
- Description updated (diff)
Updated by Peter Amstutz over 5 years ago
- Related to Feature #13493: Federated record retrieval added
Updated by Peter Amstutz over 5 years ago
- Related to Feature #13994: [Keepstore] Fetch blocks from federated clusters added
Updated by Tom Morris over 5 years ago
- Target version changed from To Be Groomed to Arvados Future Sprints
- Story points set to 2.0
Updated by
Updated by Tom Morris over 5 years ago
- Target version changed from Arvados Future Sprints to 2018-09-05 Sprint
Updated by Peter Amstutz over 5 years ago
- Related to Feature #14087: [controller] Fetch remote-hosted collections by PDH added
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2018-09-05 Sprint to 2018-09-19 Sprint
Updated by Peter Amstutz over 5 years ago
- Subject changed from [API] Fetch remote-hosted collection by UUID to [API] Fetch remote-hosted collection by UUID
- Assigned To set to Peter Amstutz
Updated by Peter Amstutz over 5 years ago
13993-federated-collection @ 9c1f240500cca97fe986b529fcfc9c7f9fe1a283
https://ci.curoverse.com/view/Developer/job/developer-run-tests/877/
Updated by Lucas Di Pentima over 5 years ago
Some comments:
- On file
lib/controller/federation.go- Line 96 - Small detail: I think just multiplying by 1.1 would more readable
- Line 76: Docs about locator format talks about lowercase hex digits (https://doc.arvados.org/api/storage.html & https://dev.arvados.org/projects/arvados/wiki/Keep_locator_format), should we stick to that definition?
- I think it would be convenient to update the locator format documentation to describe a
+Rhint - We have a previous signed locator Regex on
sdk/go/keepclient/perms.go:68that looks more correct (because I believe the current one doesn’t allow+Ahints that aren’t signature hints), should we export it so it can be reused here? - Tests are failing
Updated by Peter Amstutz over 5 years ago
Lucas Di Pentima wrote:
Some comments:
- On file
lib/controller/federation.go
- Line 96 - Small detail: I think just multiplying by 1.1 would more readable
I was trying to avoid converting to double precision and back but I agree that's a silly micro-optimization in this case at the expense of readability
- Line 76: Docs about locator format talks about lowercase hex digits (https://doc.arvados.org/api/storage.html & https://dev.arvados.org/projects/arvados/wiki/Keep_locator_format), should we stick to that definition?
See below
- I think it would be convenient to update the locator format documentation to describe a
+Rhint
I have a documentation story to write up federation features (the task on #13493 assigned to me), maybe I can do it there?
- We have a previous signed locator Regex on
sdk/go/keepclient/perms.go:68that looks more correct (because I believe the current one doesn’t allow+Ahints that aren’t signature hints), should we export it so it can be reused here?
I consolidated the regex so the one in keepclient can be used by federation.go
- Tests are failing
Works for me... Kicked off a run here: https://ci.curoverse.com/view/Developer/job/developer-run-tests/881/
Updated by Peter Amstutz over 5 years ago
Updated by Lucas Di Pentima over 5 years ago
Just one question: Would it be convenient to also add a test for a local collection request, so we test both sides?
Apart from that, it LGTM.
Updated by Peter Amstutz over 5 years ago
Lucas Di Pentima wrote:
Just one question: Would it be convenient to also add a test for a local collection request, so we test both sides?
Apart from that, it LGTM.
Added TestGetLocalCollection @ 9d2e1158e961801db714236c6942bd3596b867e7