Arvados

h1. Multi-cluster user database

It is sometimes desirable to share a single user database across multiple Arvados clusters. For example:

* Clusters aaaaa, bbbbb, ccccc, ddddd, eeeee are on different continents, but they use the same upstream authentication providers (ldap/google) and a given user either has access to all clusters, or none.

* A down/unreachable cluster should not prevent any user from using _other_ clusters in the group -- even if the down/unreachable cluster is the one where the user's account was initially created.

This requires some changes to login and token validation. (Currently, any given user account has a single "home cluster" that can issue or validate tokens for it.)

h2. Logging in

Each user should be able to log in to their account using any cluster, regardless of where/whether they have logged in previously.

To achieve this, the participating clusters agree on a single authoritative "login cluster" where the user accounts are stored, and the others hand off authentication and token-issuing to that cluster.

If aaaaa, bbbbb, ccccc, and ddddd designate eeeee as the login cluster:

* Login process on eeeee: unchanged

* Login process on aaaaa:

** aaaaa-workbench presents a link to aaaaa-controller's login endpoint (unchanged)

** aaaaa-controller redirects to eeeee-controller's login endpoint (leaving the "return_to" parameter alone)

** eeeee login process proceeds as usual: proxy through to RailsAPI, update the users table, redirect to aaaaa-workbench with the new token ("v2/eeeee-...")

* When aaaaa-workbench presents a token to aaaaa-controller, the token will have an "eeeee" prefix, so aaaaa-controller will ask eeeee to validate it, and cache the result

h2. Token validation

Login tokens are validated as before:

* All tokens start with "v2/eeeee-..." and are validated by calling back to eeeee

Per-container tokens are issued and validated as before:

* A container running on aaaaa has a "v2/aaaaa-..." token which is validated by checking aaaaa's local database

h2. Configuration

Non-login clusters need to know

* the login cluster's ID and API endpoint

<pre><code class="yaml">

Clusters:

  aaaaa:

    Login:

      LoginCluster: eeeee

    RemoteClusters:

      eeeee:

        Proxy: true

        Host: eeeee.arvadosapi.com

</code></pre>

h2. Distributed database

In the future, this can be extended by having all clusters in the group use a single distributed database for storing users and tokens with "eeeee" prefixes. For now, each non-master cluster will just continue to update its cached user record (if stale at token validation time) from the master cluster, and proxy update requests to the master.

h2. Migration

Admin migration tool (given an admin token for each cluster) should load user lists from all clusters in the group, and merge non-eeeee accounts into new/existing eeeee-* accounts.

* The identity_url field no longer provides upstream auth info, so we'll need to match accounts by email address.