Arvados

16312-s3-signature-v4 @ 4411f0b4e2a81f09d0ff6ff3f5e23cac5414236a -- developer-run-tests: #1861

I've tested this on pirca (the soon-to-be new playground cluster) on AWS. I swapped out the running keepstore with your provided binary. I then switched the bucket to AES-256 encryption, and was able to upload a block. In the S3 bucket, that block reports as encrypted:

Owner: sysadmin+playground
Last modified: May 15, 2020 11:22:02 AM GMT-0400
Etag: 84ab8ab52f42eac19801ea7b223dae3f
Storage class: Standard
Server-side encryption: AES-256
Size: 118.0 B
Key: 84ab8ab52f42eac19801ea7b223dae3f

I was also able to download the block again without issues. In other words, this seems to work!

Regarding the new V2Signature config, I also considered using a default like "default V4 if using a known AWS region, default V2 if specifying endpoint in config" so this change wouldn't affect people using non-AWS S3 backends at all. But defaulting to V4 across the board seems much easier to explain/understand. The most obvious non-AWS backends, Minio and Google, both accept V4 signatures.

I agree with changing the default to V4.

Although, having the config be "V2Signature: false" is a little weird, I don't know if there's any situation where you might need a V1 or V3 or V5 signature. Having the config be "SignatureType: V4" (default) with a note that "V2" is also supported might be a little clearer. (soft ask)

I was a little confused that you had introduced IAMRole to S3VolumeDriverParameters but I see now what you actually did was consolidate Keep's S3Volume struct with S3VolumeDriverParameters from the SDK.

The jenkins test failed, it appears to be a network timout in a Python test so it is almost certainly unrelated, but to be sure I resubmitted it:

developer-run-tests: #1865

LGTM with passing tests.