Idea #22680
Updated by Peter Amstutz 8 days ago
We want containers with API access to be able to access AWS services when running on AWS. The driving use case is to enable tasks running within containers to natively access organizational S3 buckets (not the Arvados Keep buckets).
I believe what we want to do here is going to be something like this:
a) the compute node's AWS role includes permission to assume one or more other roles. those other roles have the AWS permissions that should be available to the job inside the container.
b) crunch-run decides if the user is permitted to take on one of these AWS roles (however, an initial version of this might be 1 role for any user on the cluster)
c) crunch-run calls AssumeRole to get credentials for the container's role and gets back SessionToken, SecretAccessKey, and AccessKeyId.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html
d) The credentials assigned to the container are passed in when launching the container -- probably using environment variables
<pre>
$ export AWS_ACCESS_KEY_ID=ASIAIOSFODNN7EXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ export AWS_SESSION_TOKEN=AQoDYXdzEJr...<remainder of session token>
</pre>
Alternately, we could implement the container credential provider feature:
https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
I think this is just a HTTP GET response with the same format as instance metadata (?) but since this is kind of an internal AWS API we'd have to do a little light reverse engineering to make sure we gave it the correct response. The advantage of this is that it might allow for longer lived credentials, as each time the endpoint is called by the container, crunch-run could make a new AssumeRole call to get refreshed credentials.