Idea #13112
Updated by Peter Amstutz about 6 years ago
There needs to be a way to store credentials and other secrets securely for use by crunch containers. Requirements / use cases: * At runtime, crunch container must be able to get credentials to access remote resources, for example data transfer to remote object storage * Storing credentials easily visible places such as the container record, git repo, keep collection or docker image is unacceptable. * Storing unencrypted credentials directly in the Arvados Postgres database is not desirable. Homebrew schemes See note-2 for encrypting credentials are also not desirable. * We refer to credentials using a symbolic name, look them up to get the credentials * Because access of containers is based on Arvados user (+ special container token) access control to read credentials should also be based on Arvados user and/or container token. * From slides: ** All 3rd Party credentials will be stored in Vault-like secure system. Structure of a vault is key=value, ** Where vale are credentials provided by 3rd Party and key is being generated by operations portal. ** Initial proposal for generating credentials key is: <pre> <prefix><separator><institution_id><separator><program_id><separator><method_id><separator><file_type_id> tso_0001_0001_001_001 </pre> background discussion. Proposal: https://dev.arvados.org/projects/arvados/wiki/Vault Based on the proposal, the essential development tasks are: * Create secrets table in API server * Create Vault plugin that interacts enables login with Arvados API token and interacts with secrets table to determine policy granting access to secrets. * Arvados client support to work with secrets (at minimum, a command line client for reading, writing, and listing secrets which interacts with the API server and Vault) In order to integrate secrets handling into CWL, a couple of additional tasks are necessary * arvados-cwl-runner feature to indicate inputs that represent "secrets" and adjust the container request accordingly. * Crunch-run feature to access Vault and perform substitution of secret into config file or environment just-in-time, as part of container setup, prior to running container.