Feature #16312

Support encrypted S3 buckets

Added by Peter Amstutz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/15/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Trying to write to an encrypted bucket gets an error "Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4."


Subtasks

Task #16412: Review 16312-s3-signature-v4ResolvedTom Clegg


Related issues

Related to Arvados Epics - Story #15962: Easy cloud installResolved04/01/202005/20/2020

Associated revisions

Revision 9a71dd94
Added by Tom Clegg over 1 year ago

Merge branch '16312-s3-signature-v4'

closes #16312

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz over 1 year ago

#2 Updated by Peter Amstutz over 1 year ago

  • Description updated (diff)

#3 Updated by Peter Amstutz over 1 year ago

  • Target version set to 2020-05-20 Sprint

#4 Updated by Ward Vandewege over 1 year ago

  • Blocked by Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client library added

#5 Updated by Tom Clegg over 1 year ago

  • Assigned To set to Tom Clegg

#6 Updated by Tom Clegg over 1 year ago

  • Status changed from New to In Progress

#7 Updated by Tom Clegg over 1 year ago

  • Blocked by deleted (Story #10477: [keepstore] switch s3 driver from goamz to a more actively maintained client library)

#9 Updated by Ward Vandewege over 1 year ago

I've tested this on pirca (the soon-to-be new playground cluster) on AWS. I swapped out the running keepstore with your provided binary. I then switched the bucket to AES-256 encryption, and was able to upload a block. In the S3 bucket, that block reports as encrypted:

Owner: sysadmin+playground
Last modified: May 15, 2020 11:22:02 AM GMT-0400
Etag: 84ab8ab52f42eac19801ea7b223dae3f
Storage class: Standard
Server-side encryption: AES-256
Size: 118.0 B
Key: 84ab8ab52f42eac19801ea7b223dae3f

I was also able to download the block again without issues. In other words, this seems to work!

#10 Updated by Tom Clegg over 1 year ago

Regarding the new V2Signature config, I also considered using a default like "default V4 if using a known AWS region, default V2 if specifying endpoint in config" so this change wouldn't affect people using non-AWS S3 backends at all. But defaulting to V4 across the board seems much easier to explain/understand. The most obvious non-AWS backends, Minio and Google, both accept V4 signatures.

#11 Updated by Peter Amstutz over 1 year ago

I agree with changing the default to V4.

Although, having the config be "V2Signature: false" is a little weird, I don't know if there's any situation where you might need a V1 or V3 or V5 signature. Having the config be "SignatureType: V4" (default) with a note that "V2" is also supported might be a little clearer. (soft ask)

I was a little confused that you had introduced IAMRole to S3VolumeDriverParameters but I see now what you actually did was consolidate Keep's S3Volume struct with S3VolumeDriverParameters from the SDK.

The jenkins test failed, it appears to be a network timout in a Python test so it is almost certainly unrelated, but to be sure I resubmitted it:

https://ci.arvados.org/view/Developer/job/developer-run-tests/1865/

LGTM with passing tests.

#12 Updated by Anonymous over 1 year ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#14 Updated by Peter Amstutz about 1 year ago

  • Release set to 25

Also available in: Atom PDF