Bug #17590

[configuration] EC2 credentials variables are not consistently named in the /etc/arvados/config.yml file

Added by Javier BĂ©rtoli 15 days ago. Updated 3 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
05/06/2021
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

When setting up a new cluster in AWS I noticed the credentials variables for the Containers:CloudVMs:DriverParameters and Volumes:<volume>:DriverParameters sections are named inconsistently. This leads to involuntary, hard to debug, errors when creating the new config:

    Containers:
      CloudVMs:
        Driver: ec2
        DriverParameters:
          AccessKeyID: AKIA...5
          SecretAccessKey: 5x...MV

    Volumes:
      <cluster>-nyw5e-0000000000000000:
        DriverParameters:
          AccessKey: AKIA...5
          SecretKey: 5x...MV

The names of these variables should be consistent across the file.


Subtasks

Task #17639: Review 17590-ec2-config-keysResolvedTom Clegg

Associated revisions

Revision 19f96717
Added by Tom Clegg 6 days ago

Merge branch '17590-ec2-config-keys'

fixes #17590

Arvados-DCO-1.1-Signed-off-by: Tom Clegg <>

History

#1 Updated by Peter Amstutz 15 days ago

  • Target version changed from 2021-04-28 bughunt sprint to 2021-05-12 sprint

#2 Updated by Tom Clegg 15 days ago

  • Assigned To set to Tom Clegg

#4 Updated by Tom Clegg 8 days ago

  • Status changed from New to In Progress

#6 Updated by Ward Vandewege 6 days ago

Tom Clegg wrote:

17590-ec2-config-keys @ cef860fc944f66b4b52aa2ea6d66561944a7c065 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2459/

  • I think specifying the new keys while keeping the old ones around should be an error.
  • Can you add a note to the upgrade guide?
  • The config-check command says "deprecated or unknown config entry" but it sure would be nice if that could also give the hing that they were simply renamed. Does it only do that when Keepstore starts perhaps? This is what I see on tordo:
tordo:~# ./arvados-server config-check
time="2021-05-07T13:52:12Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.API.RailsSessionSecretToken" 
time="2021-05-07T13:52:12Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.Volumes.tordo-nyw5e-000000000000000.DriverParameters.SecretKey" 
time="2021-05-07T13:52:12Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.Volumes.tordo-nyw5e-000000000000000.DriverParameters.AccessKey" 

Otherwise, LGTM, thanks!

#7 Updated by Tom Clegg 6 days ago

Oops, it turns out the logs were unhelpful because the migration code wasn't running at all -- it was looking for driver=s3 (as spelled in the example config file) instead of driver=S3 (the spelling that actually works to select the S3 driver). With that fixed (in both migration and example config), config-check does this on tordo:

Your configuration is relying on deprecated entries. Suggest making the following changes.
--- without-deprecated-configs
+++ relying-on-deprecated-configs
@@ -495,10 +495,10 @@
             ReadOnly: false
         Driver: S3
         DriverParameters:
-          AccessKey: AAAAAAAAAAAAAAAAAAAA
+          AccessKeyID: AAAAAAAAAAAAAAAAAAAA
           Bucket: tordo-nyw5e-000000000000000-volume
           Region: us-east-1
-          SecretKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+          SecretAccessKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
         ReadOnly: false
         Replication: 2
         StorageClasses: null
time="2021-05-07T14:19:53Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.Volumes.tordo-nyw5e-000000000000000.DriverParameters.SecretKey" 
time="2021-05-07T14:19:53Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.Volumes.tordo-nyw5e-000000000000000.DriverParameters.AccessKey" 
time="2021-05-07T14:19:53Z" level=warning msg="deprecated or unknown config entry: Clusters.tordo.API.RailsSessionSecretToken" 
time="2021-05-07T14:19:53Z" level=warning msg="using your old config keys tordo.Volumes.tordo-nyw5e-000000000000000.DriverParameters.AccessKey/SecretKey -- but you should rename them to AccessKeyID/SecretAccessKey" 

Also made it an error to supply both old and new config keys, and added a note to the "upgrading" doc page.

17590-ec2-config-keys @ 659fa8bfd2f9820a9b7f955e9e9e63a1cc4275a1 -- https://ci.arvados.org/view/Developer/job/developer-run-tests/2463/

#9 Updated by Tom Clegg 6 days ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

Also available in: Atom PDF