Project

General

Profile

Actions
Copy link

Feature #15599

closed

[keepstore] AWS support IAM roles for authentication

Added by Ward Vandewege over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Tom Clegg
Category:
-
Target version:
2019-10-09 Sprint
Story points:
2.0
Release:
Arvados 2.0 - Early 2020
Release relationship:
Auto

Description

AWS best practice for services running on EC2 is to use IAM roles for authentication. The AWS go sdk supports this natively, cf. https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

It would be nice if Keepstore supported IAM roles, perhaps falling back to that authentication method when SecretKeyFile and AccessKeyFile are not supplied in the configuration file. Maybe using the metadata to detect that it's running on EC2 first so we don't provide confusing information when no credentials are present and Keepstore is running elsewhere.

Getting credentials from instance metadata: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

Subtasks 1 (0 open — 1 closed)

Task #15619: Review 15599-keepstore-iam-roleResolvedPeter Amstutz10/01/2019Actions
Actions
Copy link
 #1

Updated by Ward Vandewege over 5 years ago

  • Target version set to To Be Groomed
Actions
Copy link
 #2

Updated by Tom Clegg over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Tom Morris over 5 years ago

  • Target version changed from To Be Groomed to Arvados Future Sprints
  • Story points set to 2.0
Actions
Copy link
 #4

Updated by Tom Morris over 5 years ago

  • Target version changed from Arvados Future Sprints to 2019-09-25 Sprint
Actions
Copy link
 #5

Updated by Tom Clegg over 5 years ago

  • Assigned To set to Tom Clegg
Actions
Copy link
 #9

Updated by Tom Clegg over 5 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #10

Updated by Tom Clegg over 5 years ago

15599-keepstore-iam-role @ 47afcd7595b49cf8a1756cb8f00139cd6269f544 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1521/

(hasn't been tested on a real cluster yet)

Actions
Copy link
 #12

Updated by Tom Clegg over 5 years ago

  • Target version changed from 2019-09-25 Sprint to 2019-10-09 Sprint
Actions
Copy link
 #13

Updated by Tom Clegg over 5 years ago

Tested on 4xphq with explicit role name in config file:

{"PID":18718,"level":"info","msg":"keepstore dev starting, pid 18718","time":"2019-09-27T21:00:49.465594064Z"}
{"PID":18718,"URL":"http://169.254.169.254/latest/meta-data/iam/security-credentials/keepstore-s3","level":"debug","msg":"getting credentials","time":"2019-09-27T21:00:49.465725839Z"}
{"AccessKeyID":"ASIA3EZBYHRBBUPFAKFB","Expiration":"2019-09-28T02:57:38Z","LastUpdated":"2019-09-27T20:43:26Z","PID":18718,"TTL":"5h51m48.533271563s","level":"debug","msg":"updated credentials","time":"2019-09-27T21:00:49.466734253Z"}
{"PID":18718,"level":"info","msg":"started volume 4xphq-nyw5e-dk9mjspdg2v8mhq (s3-bucket:\"4xphq-keep\"), ReadOnly=false","time":"2019-09-27T21:00:49.466829958Z"}
{"Listen":"10.20.65.1:22222","PID":18718,"Service":"keepstore","URL":"http://keep0.4xphq.arvadosapi.com:22222","level":"info","msg":"listening","time":"2019-09-27T21:00:49.483401771Z"}
{"PID":18718,"RequestID":"req-1r4cxk9sf6cua1xnv358","level":"info","msg":"request","remoteAddr":"10.20.65.1:27940","reqBytes":3,"reqForwardedFor":"","reqHost":"keep0:22222","reqMethod":"PUT","reqPath":"acbd18db4cc2f85cedef654fccc4a4d8","reqQuery":"","time":"2019-09-27T21:00:55.469119263Z"}
{"PID":18718,"RequestID":"req-1r4cxk9sf6cua1xnv358","level":"info","msg":"response","remoteAddr":"10.20.65.1:27940","reqBytes":3,"reqForwardedFor":"","reqHost":"keep0:22222","reqMethod":"PUT","reqPath":"acbd18db4cc2f85cedef654fccc4a4d8","reqQuery":"","respBytes":86,"respStatus":"OK","respStatusCode":200,"time":"2019-09-27T21:00:55.600618560Z","timeToStatus":0.131478,"timeTotal":0.131497,"timeWriteBody":0.000019}

Tested on 4xphq with no role/keys at all in config file:

{"PID":22175,"level":"info","msg":"keepstore dev starting, pid 22175","time":"2019-09-27T21:17:04.277819697Z"}
{"PID":22175,"URL":"http://169.254.169.254/latest/meta-data/iam/security-credentials/","level":"debug","msg":"looking up IAM role name","time":"2019-09-27T21:17:04.277925380Z"}
{"PID":22175,"Role":"keepstore-s3","level":"debug","msg":"looked up IAM role name","time":"2019-09-27T21:17:04.278761723Z"}
{"PID":22175,"URL":"http://169.254.169.254/latest/meta-data/iam/security-credentials/keepstore-s3","level":"debug","msg":"getting credentials","time":"2019-09-27T21:17:04.278789657Z"}
{"AccessKeyID":"ASIA3EZBYHRBBUPFAKFB","Expiration":"2019-09-28T02:57:38Z","LastUpdated":"2019-09-27T20:43:26Z","PID":22175,"TTL":"5h35m33.720482749s","level":"debug","msg":"updated credentials","time":"2019-09-27T21:17:04.279522613Z"}
{"PID":22175,"level":"info","msg":"started volume 4xphq-nyw5e-dk9mjspdg2v8mhq (s3-bucket:\"4xphq-keep\"), ReadOnly=false","time":"2019-09-27T21:17:04.279577157Z"}
{"Listen":"10.20.65.1:22222","PID":22175,"Service":"keepstore","URL":"http://keep0.4xphq.arvadosapi.com:22222","level":"info","msg":"listening","time":"2019-09-27T21:17:04.290799872Z"}
{"PID":22175,"RequestID":"req-1ozvvwrenud7l1kn6t7i","level":"info","msg":"request","remoteAddr":"10.20.65.1:28180","reqBytes":3,"reqForwardedFor":"","reqHost":"keep0:22222","reqMethod":"PUT","reqPath":"acbd18db4cc2f85cedef654fccc4a4d8","reqQuery":"","time":"2019-09-27T21:18:01.033740317Z"}
{"PID":22175,"RequestID":"req-1ozvvwrenud7l1kn6t7i","level":"info","msg":"response","remoteAddr":"10.20.65.1:28180","reqBytes":3,"reqForwardedFor":"","reqHost":"keep0:22222","reqMethod":"PUT","reqPath":"acbd18db4cc2f85cedef654fccc4a4d8","reqQuery":"","respBytes":86,"respStatus":"OK","respStatusCode":200,"time":"2019-09-27T21:18:01.172539408Z","timeToStatus":0.138775,"timeTotal":0.138793,"timeWriteBody":0.000019}

15599-keepstore-iam-role @ 0b4def357b45cec17ee45673aa14d64ec99c56c3 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1558/

Actions
Copy link
 #16

Updated by Peter Amstutz over 5 years ago

Documentation nit, the example in configure-s3-object-storage.html has values for both "IAMRole" and "AccessKey"/"SecretKey", which are mutually exclusive. (Is there are error if they are both configured?) Would be better to leave all the fields blank and explain the default behavior and then how to override it.

Actions
Copy link
 #17

Updated by Tom Clegg over 5 years ago

15599-keepstore-iam-role @ fbbba2116c046a3c20a71ea0268501a1a5b802e5 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1573/
  • config example has blanks, and presents blank/automatic as the normal case
  • startup error if config is ambiguous (IAMRole and Secret/AccessKey are both non-empty)
Actions
Copy link
 #18

Updated by Peter Amstutz over 5 years ago

Tom Clegg wrote:

15599-keepstore-iam-role @ fbbba2116c046a3c20a71ea0268501a1a5b802e5 -- https://ci.curoverse.com/view/Developer/job/developer-run-tests/1573/
  • config example has blanks, and presents blank/automatic as the normal case
  • startup error if config is ambiguous (IAMRole and Secret/AccessKey are both non-empty)

LGTM

Actions
Copy link
 #19

Updated by Tom Clegg over 5 years ago

  • Status changed from In Progress to Resolved
Actions
Copy link
 #20

Updated by Tom Clegg over 5 years ago

  • Release set to 22
Actions
Copy link

Also available in: Atom PDF