Arvados

h1. Credential storage

h2. Background

In order to implement [[Objects as pseudo-blocks in Keep]], we need a way to store credentials so that Arvados can authenticate to other systems, e.g. AWS S3.

The current system for managing secrets is specific to workflows and deletes the secret as soon as the workflow is finished.  However, we require a credential storage system that can be accessed by keepstore.

User perspective:

Want to be able to manage credentials in workbench, and then Arvados services that need it can look it them up.  The motivating use case is AWS credentials that have a key id/key secret pair (much like arvados API key uuid / secret) so that we can easily access objects in external S3 buckets.

h2. Requirements

* Secrets should have an id for what type of thing they are, e.g. AWS credentials

* Secrets should have an optional scope.  E.g. want to be able to provide different credentials for different resources, buckets, etc.

* Should the secret material itself be simple text column or a JSON object?  For example AWS secret id/secret is a pair

* Different users should have different views of what secrets are available based on Arvados permissions.  User should be able to share secrets at different levels of access, e.g.

** can_read -- system services can fetch the credential on behalf of the user, but they cannot fetch it directly through the API

** can_write -- user can update the credential, but still not read it back

** can_manage -- user can grant permissions to the credential, but still not read it back

* Secrets should be write-only as much as possible, system services can retrieve secrets, but users cannot except in special circumstances

** want a way to use secrets in workflows, which means they can be exposed if developers are careless.  This is true of our current secrets support as well (it's inherently impossible to prevent it from being leaked in user-provided code if someone is really trying, but we'll at least be able to keep a record of which workflows accessed those secrets).