Story #13146

[API] Endpoint to get projects shared with me

Added by Peter Amstutz over 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
08/15/2018
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0
Release:
Release relationship:
Auto

Description

To get "projects shared with me" efficiently requires a new API endpoint. The way arv-mount currently determines what to list in "shared" currently requires looking at all projects and finding the ones where owner_uuid is not another project which is visible to us (meaning: users, non-project groups, or shared subprojects where the parent is not visible). This is expensive to compute on
the client, but can probably be accomplished with a single query on the API server.

Proposed endpoint: /arvados/v1/groups/shared

Return in "items" a list of toplevel projects and collections which are shared with me. Toplevel projects are ones which are either owned by a user, owned by a non-project group, or owned by a group which the user cannot read.

Support "includes" to return the set of owner objects as well (users, groups).

Possible query (abbreviated):

  • select groups that are readable by current user AND
    • the owner_uuid is a user (not a group) OR
    • the owner_uuid is a group is not a project or is not readable by the current user
select uuid from groups where 
  exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.uuid) and 
  (groups.owner_uuid in (select uuid from users) or 
    not exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.owner_uuid)) and 
  owner_uuid != '2tlax-tpzed-51vcbmac4uv6bpb';

Subtasks

Task #13950: Review 13146-shared-railsResolvedPeter Amstutz

Task #14060: shared endpointResolvedPeter Amstutz


Related issues

Related to Arvados - Bug #12990: [FUSE] Access shared/ is inefficientIn Progress

Related to Arvados - Story #13111: [WebDAV] Support browsing of project hierarchiesResolved03/27/2018

Related to Arvados - Story #13218: Support browsing of projects shared with me in WebDAVNew

Related to Arvados - Feature #14201: [API] Accept "exclude_home_project" flag in groups#contentsResolved10/01/2018

Related to Arvados Workbench 2 - Support #15238: [Workbench] Change the content inside the owner column from uuid to Projects and user namesNew

Associated revisions

Revision 6d6a0b0f
Added by Peter Amstutz almost 2 years ago

Merge branch '13146-shared-rails' refs #13146

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz over 2 years ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz over 2 years ago

  • Description updated (diff)

#3 Updated by Peter Amstutz over 2 years ago

  • Related to Bug #12990: [FUSE] Access shared/ is inefficient added

#4 Updated by Peter Amstutz over 2 years ago

  • Related to Story #13111: [WebDAV] Support browsing of project hierarchies added

#5 Updated by Tom Morris over 2 years ago

  • Tracker changed from Feature to Story
  • Status changed from In Progress to New

#6 Updated by Tom Morris over 2 years ago

  • Related to Story #13218: Support browsing of projects shared with me in WebDAV added

#7 Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)

#8 Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)

The current algorithm used in the FUSE driver is:

  • get all projects
  • go through the list of projects and find projects which have an owner_uuid that isn't in the list and isn't the current user. This has the effect of finding the projects for which the owner is either a user, a group which is not a project, or a group which is not readable by the current user.
  • The list of "roots" consists of each owner (if the owner is readable but not a project) or the project itself (if not).

#9 Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 3.0

#10 Updated by Tom Morris almost 2 years ago

  • Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

#11 Updated by Peter Amstutz almost 2 years ago

  • Assigned To set to Peter Amstutz
  • Target version changed from 2018-08-15 Sprint to Arvados Future Sprints

#12 Updated by Peter Amstutz almost 2 years ago

  • Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

#13 Updated by Peter Amstutz almost 2 years ago

  • Status changed from New to In Progress

#14 Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2018-08-15 Sprint to 2018-09-05 Sprint

#15 Updated by Peter Amstutz almost 2 years ago

13146-shared-rails @ 0ba8a53ba145475153b01dc498f85dbd2f03228b

https://ci.curoverse.com/view/Developer/job/developer-run-tests/850/

I spent a while mulling over alternate APIs for "shared" but kept coming back the original definition. It is a bit ugly but it should still be much more efficient than listing every project and user and working it out on the client side.

#16 Updated by Tom Clegg almost 2 years ago

Suggestions

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

#18 Updated by Peter Amstutz almost 2 years ago

Tom Clegg wrote:

Suggestions

Done.

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

See previous note.

Now 13146-shared-rails @ a41c0f6aa41b658c8f2947c46cb90778894f5cf3

https://ci.curoverse.com/view/Developer/job/developer-run-tests/855/

#19 Updated by Tom Clegg almost 2 years ago

Experiments & include/included changes LGTM.

In the docs...
  • The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".
  • It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

#20 Updated by Peter Amstutz almost 2 years ago

Tom Clegg wrote:

Experiments & include/included changes LGTM.

In the docs...
  • The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".
  • It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

Sure, documentation updated @ 489aed58cb0d8bd816e07128cfcb9f5a06224083

#21 Updated by Tom Clegg almost 2 years ago

LGTM, thanks.

(Nit: seems like "this is useful for clients which wish to browse the list of" could be replaced with "i.e.," since it's implicit that an API is useful for clients that wish to get the information it returns... but this is gold-plated enough and I'm ready to move on)

#22 Updated by Tom Morris almost 2 years ago

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

#23 Updated by Peter Amstutz almost 2 years ago

  • Status changed from In Progress to Resolved

Tom Morris wrote:

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

This is done. The workbench2 folks should be able to use it (provided they are accessing one of the dev clusters with the latest code).

#24 Updated by Peter Amstutz almost 2 years ago

And it's documented at the bottom of the page here:

http://doc.arvados.org/api/methods/groups.html

#25 Updated by Tom Clegg almost 2 years ago

  • Related to Feature #14201: [API] Accept "exclude_home_project" flag in groups#contents added

#26 Updated by Ward Vandewege almost 2 years ago

  • Release set to 13

#27 Updated by Tom Morris about 1 year ago

  • Related to Support #15238: [Workbench] Change the content inside the owner column from uuid to Projects and user names added

Also available in: Atom PDF