Arvados

There needs to be a way to store credentials and other secrets securely for use by crunch containers.

Requirements / use cases:

• At runtime, crunch container must be able to get credentials to access remote resources, for example data transfer to remote object storage
• Storing credentials easily visible places such as the container record, git repo, keep collection or docker image is unacceptable.
• Storing unencrypted credentials directly in the Arvados Postgres database is not desirable. Homebrew schemes for encrypting credentials are also not desirable.
• We refer to credentials using a symbolic name, look them up to get the credentials
• Because access of containers is based on Arvados user (+ special container token) access control to read credentials should also be based on Arvados user and/or container token.
• From slides:
  • All 3rd Party credentials will be stored in Vault-like secure system. Structure of a vault is key=value,
  • Where vale are credentials provided by 3rd Party and key is being generated by operations portal.
  • Initial proposal for generating credentials key is:
    

    <prefix><separator><institution_id><separator><program_id><separator><method_id><separator><file_type_id>
    tso_0001_0001_001_001
    

Proposal:

https://dev.arvados.org/projects/arvados/wiki/Vault

Based on the proposal, the essential development tasks are:

• Create secrets table in API server
• Create Vault plugin that interacts enables login with Arvados API token and interacts with secrets table to determine policy granting access to secrets.
• Arvados client support to work with secrets (at minimum, a command line client for reading, writing, and listing secrets which interacts with the API server and Vault)

In order to integrate secrets handling into CWL, a couple of additional tasks are necessary

• arvados-cwl-runner feature to indicate inputs that represent "secrets" and adjust the container request accordingly.
• Crunch-run feature to access Vault and perform substitution of secret into config file or environment just-in-time, as part of container setup, prior to running container.