Idea #13146: [API] Endpoint to get projects shared with me - Arvados

Custom queries

All assigned issues
All issues assigned for grooming
My issues for grooming
My issues for grooming (no story pts)
Prioritized open issues

Actions

Copy link

Idea #13146

closed

[API] Endpoint to get projects shared with me

Added by Peter Amstutz almost 7 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Peter Amstutz

Category:

Target version:

2018-09-05 Sprint

Start date:

08/15/2018

Due date:

Story points:

3.0

Release:

Arvados 1.2

Release relationship:

Auto

Description

To get "projects shared with me" efficiently requires a new API endpoint. The way arv-mount currently determines what to list in "shared" currently requires looking at all projects and finding the ones where owner_uuid is not another project which is visible to us (meaning: users, non-project groups, or shared subprojects where the parent is not visible). This is expensive to compute on
the client, but can probably be accomplished with a single query on the API server.

Proposed endpoint: /arvados/v1/groups/shared

Return in "items" a list of toplevel projects and collections which are shared with me. Toplevel projects are ones which are either owned by a user, owned by a non-project group, or owned by a group which the user cannot read.

Support "includes" to return the set of owner objects as well (users, groups).

Possible query (abbreviated):

select groups that are readable by current user AND
- the owner_uuid is a user (not a group) OR
- the owner_uuid is a group is not a project or is not readable by the current user

select uuid from groups where 
  exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.uuid) and 
  (groups.owner_uuid in (select uuid from users) or 
    not exists(select 1 from materialized_permission_view where user_uuid='2tlax-tpzed-51vcbmac4uv6bpb' and target_uuid=groups.owner_uuid)) and 
  owner_uuid != '2tlax-tpzed-51vcbmac4uv6bpb';

Subtasks 2 (0 open — 2 closed)

	Task #13950: Review 13146-shared-rails	Resolved	Peter Amstutz	08/16/2018			Actions
	Task #14060: shared endpoint	Resolved	Peter Amstutz	08/15/2018			Actions

Related issues 5 (1 open — 4 closed)

Related to Arvados - Bug #12990: [FUSE] Access shared/ is inefficient	Resolved	Peter Amstutz		Actions
Related to Arvados - Idea #13111: [WebDAV] Support browsing of project hierarchies	Resolved	Tom Clegg	03/27/2018	Actions
Related to Arvados - Idea #13218: Support browsing of projects shared with me in WebDAV	New			Actions
Related to Arvados - Feature #14201: [API] Accept "exclude_home_project" flag in groups#contents	Resolved	Peter Amstutz	10/01/2018	Actions
Related to Arvados Workbench 2 - Support #15238: [Workbench] Change the content inside the owner column from uuid to Projects and user names	Resolved			Actions

Issue # Delay: days Cancel

History
Notes
Property changes
Associated revisions

Actions

Copy link

Updated by Peter Amstutz almost 7 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Peter Amstutz almost 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz almost 7 years ago

Related to Bug #12990: [FUSE] Access shared/ is inefficient added

Actions

Copy link

Updated by Peter Amstutz almost 7 years ago

Related to Idea #13111: [WebDAV] Support browsing of project hierarchies added

Actions

Copy link

Updated by Tom Morris almost 7 years ago

Tracker changed from Feature to Idea
Status changed from In Progress to New

Actions

Copy link

Updated by Tom Morris almost 7 years ago

Related to Idea #13218: Support browsing of projects shared with me in WebDAV added

Actions

Copy link

Updated by Peter Amstutz over 6 years ago

Description updated (diff)

Actions

Copy link

Updated by Peter Amstutz over 6 years ago

Description updated (diff)

The current algorithm used in the FUSE driver is:

get all projects
go through the list of projects and find projects which have an owner_uuid that isn't in the list and isn't the current user. This has the effect of finding the projects for which the owner is either a user, a group which is not a project, or a group which is not readable by the current user.
The list of "roots" consists of each owner (if the owner is readable but not a project) or the project itself (if not).

Actions

Copy link

Updated by Peter Amstutz over 6 years ago

Description updated (diff)
Target version changed from To Be Groomed to Arvados Future Sprints
Story points set to 3.0

Actions

Copy link

#10

Updated by Tom Morris over 6 years ago

Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

Actions

Copy link

#11

Updated by Peter Amstutz over 6 years ago

Assigned To set to Peter Amstutz
Target version changed from 2018-08-15 Sprint to Arvados Future Sprints

Actions

Copy link

#12

Updated by Peter Amstutz over 6 years ago

Target version changed from Arvados Future Sprints to 2018-08-15 Sprint

Actions

Copy link

#13

Updated by Peter Amstutz over 6 years ago

Status changed from New to In Progress

Actions

Copy link

#14

Updated by Peter Amstutz over 6 years ago

Target version changed from 2018-08-15 Sprint to 2018-09-05 Sprint

Actions

Copy link

#15

Updated by Peter Amstutz over 6 years ago

13146-shared-rails @ 0ba8a53ba145475153b01dc498f85dbd2f03228b

https://ci.curoverse.com/view/Developer/job/developer-run-tests/850/

I spent a while mulling over alternate APIs for "shared" but kept coming back the original definition. It is a bit ugly but it should still be much more efficient than listing every project and user and working it out on the client side.

Actions

Copy link

#16

Updated by Tom Clegg over 6 years ago

Suggestions

rename the "include" response field to "included" like http://jsonapi.org/format/#document-compound-documents
...and rename "includes" to "included" in the comment in shared()
skip the owners unless the client asks for them with ?include=owner like http://jsonapi.org/format/#fetching-includes
apply @select to the related objects as well

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

Actions

Copy link

#18

Updated by Peter Amstutz over 6 years ago

Tom Clegg wrote:

Suggestions

rename the "include" response field to "included" like http://jsonapi.org/format/#document-compound-documents

...and rename "includes" to "included" in the comment in shared()

skip the owners unless the client asks for them with ?include=owner like http://jsonapi.org/format/#fetching-includes>

apply @select to the related objects as well

Done.

Have you tried these queries on a production-size database to make sure performance is OK with the existing indices?

See previous note.

Now 13146-shared-rails @ a41c0f6aa41b658c8f2947c46cb90778894f5cf3

https://ci.curoverse.com/view/Developer/job/developer-run-tests/855/

Actions

Copy link

#19

Updated by Tom Clegg over 6 years ago

Experiments & include/included changes LGTM.

In the docs...

The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".
It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

Actions

Copy link

#20

Updated by Peter Amstutz over 6 years ago

Tom Clegg wrote:

Experiments & include/included changes LGTM.
In the docs...

The second paragraph should have some reference to the "include" flag, perhaps just the word "optionally" -- as written it implies "always".

It feels like we're saying the same thing 4 different ways, including some pseudocode that has "a and b or c or d", which makes me think about operator precedence, and an "intended use" paragraph that seems to amount to "this API exists to support clients which wish to do what this API does". Could we pare this back to one or two ways of explaining it?

Sure, documentation updated @ 489aed58cb0d8bd816e07128cfcb9f5a06224083

Actions

Copy link

#21

Updated by Tom Clegg over 6 years ago

LGTM, thanks.

(Nit: seems like "this is useful for clients which wish to browse the list of" could be replaced with "i.e.," since it's implicit that an API is useful for clients that wish to get the information it returns... but this is gold-plated enough and I'm ready to move on)

Actions

Copy link

#22

Updated by Tom Morris over 6 years ago

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

Actions

Copy link

#23

Updated by Peter Amstutz over 6 years ago

Status changed from In Progress to Resolved

Tom Morris wrote:

It looks like a branch got merged a few days ago, but this ticket is still marked as being "In Progress." Is there additional work to be done? We were looking at it during the Workbench2 review to see what it's status was.

This is done. The workbench2 folks should be able to use it (provided they are accessing one of the dev clusters with the latest code).

Actions

Copy link

#24