Support encrypted S3 buckets
Trying to write to an encrypted bucket gets an error "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4."
#8 Updated by Tom Clegg over 1 year ago
16312-s3-signature-v4 @ 4411f0b4e2a81f09d0ff6ff3f5e23cac5414236a -- https://ci.arvados.org/view/Developer/job/developer-run-tests/1861/
#9 Updated by Ward Vandewege over 1 year ago
I've tested this on pirca (the soon-to-be new playground cluster) on AWS. I swapped out the running keepstore with your provided binary. I then switched the bucket to AES-256 encryption, and was able to upload a block. In the S3 bucket, that block reports as encrypted:
Owner: sysadmin+playground Last modified: May 15, 2020 11:22:02 AM GMT-0400 Etag: 84ab8ab52f42eac19801ea7b223dae3f Storage class: Standard Server-side encryption: AES-256 Size: 118.0 B Key: 84ab8ab52f42eac19801ea7b223dae3f
I was also able to download the block again without issues. In other words, this seems to work!
#10 Updated by Tom Clegg over 1 year ago
Regarding the new V2Signature config, I also considered using a default like "default V4 if using a known AWS region, default V2 if specifying endpoint in config" so this change wouldn't affect people using non-AWS S3 backends at all. But defaulting to V4 across the board seems much easier to explain/understand. The most obvious non-AWS backends, Minio and Google, both accept V4 signatures.
#11 Updated by Peter Amstutz over 1 year ago
I agree with changing the default to V4.
Although, having the config be "V2Signature: false" is a little weird, I don't know if there's any situation where you might need a V1 or V3 or V5 signature. Having the config be "SignatureType: V4" (default) with a note that "V2" is also supported might be a little clearer. (soft ask)
I was a little confused that you had introduced IAMRole to S3VolumeDriverParameters but I see now what you actually did was consolidate Keep's S3Volume struct with S3VolumeDriverParameters from the SDK.
The jenkins test failed, it appears to be a network timout in a Python test so it is almost certainly unrelated, but to be sure I resubmitted it:
LGTM with passing tests.