Multi-cluster user database¶

It is sometimes desirable to share a single user database across multiple Arvados clusters. For example:

• Clusters aaaaa, bbbbb, ccccc are on different continents.
• A down/unreachable cluster should not prevent a user from accessing other clusters -- even if the down/unreachable cluster is normally the best/default one from that user's perspective.

This requires some changes to authentication (obtaining and validating API tokens).

Obtaining tokens¶

Each user should be able to log in to their account using any cluster, regardless of where/whether they have logged in previously. This contrasts with the current setup, where each user account has a "home cluster" which must be used to log in.

To achieve this (without depending real-time communication between clusters) we need all of the participating clusters to agree on a mapping of upstream authentication results to Arvados user UUIDs. For example, if the upstream authentication result is "ldap://ldap.example foo@bar.example" ("ldap://ldap.example assures us this user is foo@bar.example"):

1. If a row already exists in the users table with upstream == "ldap://ldap.example foo@bar.example" then use that row
2. Otherwise, create a new row with user UUID "eeeee-tpzed-${sha1part(upstream)}" (where eeeee is a common prefix used by all participating clusters and sha1part() is the first 15 chars of base-36-encoded sha1())

To avoid changing existing user accounts' UUIDs to eeeee-*, we would do a one-time synchronization of user accounts (and their upstreams) across all participating clusters. For example, if aaaaa-tpzed-012340123401234 exists on cluster aaaaa, we would add that row to bbbbb and ccccc as well. Next time a user logs in to bbbbb with an upstream account matching aaaaa-tpzed-012340123401234, bbbbb would issue a token itself, rather than deferring to aaaaa.

Untrusted remote accounts (the kind that we already have in the users table with foreign UUIDs) have a null upstream field.

uuid	upstream	significance
aaaaa-tpzed-aaaaaaaaaaaaaaa	google:// foo@bar.example	Imported/migrated from remote cluster aaaaa
eeeee-tpzed-012340123401234	ldap://ldap.example foo@baz.example	User didn't exist before the multi-cluster user db system arrived
ooooo-tpzed-ooooooooooooooo	NULL	Remote user from cluster ooooo (not part of our multi-cluster group)

Validating tokens¶

Each cluster should be able to validate a token that was issued by a different, currently unreachable, cluster. This contrasts with the current setup, where aaaaa validates tokens issued by bbbbb by doing a callback to bbbbb.

This seems easy enough: instead of random strings, tokens can be [like] JWT, signed by a private key whose public part is known by all clusters. (This would also be more efficient than callbacks, benefiting the mutually-untrusted cluster scenario too.)

Configuration¶

Each cluster needs to know

• the uuid prefix to use when creating a new account, e.g., "eeeee"
• additional user uuid prefixes that remote clusters are trusted to validate

Clusters:
  aaaaa:
    RemoteClusters:
      bbbbb:
        Proxy: true
        Authenticate:
          aaaaa: {} # accept tokens issued by bbbbb for users with uuid aaaaa-*
          bbbbb: {} # (implied)
          eeeee: {} # accept tokens issued by bbbbb for users with uuid eeeee-*

Example: aaaaa needs to validate a token issued by bbbbb.

• Do a callback to bbbbb (or check JWT signature) to confirm bbbbb really issued this token (result: yes, user uuid is eeeee-tpzed-012340123401234)
• If config Clusters.aaaaa.RemoteClusters.bbbbb.Authenticate.eeeee is present, accept the token
• Otherwise, fetch eeeee's config; if RemoteClusters.bbbbb.Authenticate.eeeee is present, accept the token
• Otherwise, reject the token