Story #15720

[API] Unified user listing across all clusters in a federation

Added by Peter Amstutz 8 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
11/19/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
3.0
Release relationship:
Auto

Description

When choosing to share a project with another user, the contents of the dialog box is populated from the user database of the local cluster. Non-local users only appear if they have previously accessed that cluster in some way (causing a user record to be created). This means when attempting to share a project with another user, that user may or may not appear in the list based on whether the user has ever accessed that cluster in the past. (It is also invalid to create a sharing involving a user uuid that doesn't appear in the users table). This is confusing.

For the case where LoginCluster is set, this can be solved by having arvados controller proxy user record listing requests to the LoginCluster. The API server also needs to allow permission links that point to users uuids that are not already cached in the users table -- possibly by automatically creating a stub record for those users.

For the general case, controller would need to query the user listings of all known remote clusters and merge them into a single response. (we could probably defer this)

Kind of related: what happens when a user merges accounts on the main cluster? All the other clusters should be synchronized and merge the old user to the new user. Should the API server merge account feature do this?

Agreed solution for the LoginCluster case:

When LoginCluster is set, Controller proxies requests to "get" or "list" user records to the LoginCluster.

Controller uses the response from the LoginCluster to create or update user records in the local database before returning the response to the client.

If the query contains 'select' only update the fields in the response (must include 'uuid').


Subtasks

Task #15809: Review 15720-fed-user-listResolvedTom Clegg


Related issues

Related to Arvados - Feature #15531: [SDK] Migrate federation to central LoginClusterResolved09/23/2019

Related to Arvados - Story #15795: [API] Accept configured SystemRootToken without doing a database lookupResolved11/23/2019

Associated revisions

Revision c3715559
Added by Tom Clegg 6 months ago

Merge branch '15720-fed-user-list'

closes #15720

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

Revision eb0b1549 (diff)
Added by Tom Clegg 6 months ago

Update token in keepproxy test.

Fixes commit b047cc92acab06d4c1f1d80173486f951b63d729.

refs #15720

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 8 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 8 months ago

  • Related to Feature #15531: [SDK] Migrate federation to central LoginCluster added

#4 Updated by Peter Amstutz 8 months ago

  • Status changed from In Progress to New

#5 Updated by Peter Amstutz 8 months ago

  • Subject changed from Unified user listing across all clusters in a federation to [API] Unified user listing across all clusters in a federation
  • Description updated (diff)

#6 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#7 Updated by Peter Amstutz 8 months ago

  • Description updated (diff)

#8 Updated by Tom Morris 8 months ago

  • Tracker changed from Bug to Story

#9 Updated by Peter Amstutz 7 months ago

  • Description updated (diff)

#10 Updated by Peter Amstutz 7 months ago

  • Description updated (diff)
  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 3.0

#11 Updated by Peter Amstutz 7 months ago

  • Target version changed from Arvados Future Sprints to 2019-11-20 Sprint

#12 Updated by Tom Clegg 7 months ago

  • Assigned To set to Tom Clegg

#13 Updated by Tom Clegg 7 months ago

15720-fed-user-list @ b047cc92acab06d4c1f1d80173486f951b63d729 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1651/

I think this is functional, but has a flaw (which I figure is a blocker) that it involves N controller-to-RailsAPI update requests for each response page of N items. Ways to fix:
  1. do a local query with filters=[uuid,in,[...]], select=[...] to compare, then update only the items that have changed (might need multiple local queries if the local max page size is smaller than the remote max page size)
  2. add a batch-update API to RailsAPI, perhaps requiring SystemRootToken in order to avoid permission-related nuances

#14 Updated by Tom Clegg 7 months ago

  1. add a batch-update API to RailsAPI, perhaps requiring SystemRootToken in order to avoid permission-related nuances

15720-fed-user-list @ 233a2b6bd23a3e2054cfc0690f2bc06c0f9f7323 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1658/

#15 Updated by Tom Clegg 7 months ago

  • Status changed from New to In Progress

#16 Updated by Tom Clegg 7 months ago

  • Target version changed from 2019-11-20 Sprint to 2019-12-04 Sprint

#17 Updated by Peter Amstutz 6 months ago

EnableBetaController14287 needs to be on for this feature to work. Should this be the branch that enables it by default?

If you contact a cluster with a token issued by that cluster (eg the superuser token, rather than the login cluster), it won't be accepted by the login cluster and you get this:

Error: request failed: https://172.17.0.2:8000/arvados/v1/users?cluster_id=&count=exact&filters=%5B%5B%22email%22%2C%22%3D%22%2C%22a%40a%22%5D%5D&limit=100&offset=0: 401 Unauthorized: Not logged in (req-9fdv9jdhlz7b298vo3t3)

Next try, talking to cluster (B) using a token issued by cluster (A):

Error: error updating local user records: request failed: http://localhost:8004/arvados/v1/users/batch_update: 401 Unauthorized: Not logged in (req-oagajjudmev71bd4u28r)

On futher research, it looks like arvbox has fallen into one of our circular dependency setup traps. The SystemRootToken is set in the config file, you can't start the API server without a config file, so after starting the API server, I would need to create an api_client_authorization record for the system root token, rewrite the the config file and then restart controller. Uhg.

I think we discussed having API accept SystemRootToken as a special case, could we sneak this in? I really really want to stamp out all the circular dependencies between the config file and API. (Perhaps I'll just go ahead and make that branch over the weekend).

#18 Updated by Tom Clegg 6 months ago

#15795 "Accept configured SystemRootToken ..."

#19 Updated by Tom Clegg 6 months ago

  • Related to Story #15795: [API] Accept configured SystemRootToken without doing a database lookup added

#21 Updated by Peter Amstutz 6 months ago

Would it be more RESTful to PATCH on /arvados/v1/users instead of /arvados/v1/users/batch_update? In which case the format should be a list of user objects (like the index query) instead of a uuid->object map. I guess making it a special-purpose endpoint avoids these questions.

Looks like batch_update is not intended to be a public API, there's no EndpointUserBatchUpdate APIEndpoint in the Go SDK. Looks like it doesn't appear in the discovery document, so that's fine.

It is a little weird that the code generation starts from ContainerList and rewrites it for other variants instead of starting from a fully generic template.

I was very confused that new user records were appearing on one of the remote clusters even though there was no obvious user interaction. It turns out that workbench1 does session login on each remote cluster even when not on the multi-site search page, even if multi-site search is not enabled. (I expected something like this with workbench2, but not workbench1).

However, even though it turns out that workbench causes users to be automatically cached at other cluster as a side effect, the user listing is still necessary to ensure the correct user list is displayed (based on group membership) and correct user info (like first name / last name).

With that in mind, I made changes to user's group membership and user's name on the login cluster and those changes were reflected in the sharing dialog of a federate cluster's workbench. Success!

This LGTM.

#22 Updated by Tom Clegg 6 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#23 Updated by Peter Amstutz 4 months ago

  • Release set to 22

Also available in: Atom PDF