Feature #21705
open
Update Go runtime and dependencies
Added by Tom Clegg 28 days ago. Updated 1 day ago.
Files
| cloudtest.log.txt (6.08 KB) cloudtest.log.txt | Tom Clegg, 05/15/2024 02:23 PM |
Updated by Peter Amstutz 9 days ago
- Target version changed from Development 2024-05-08 sprint to Development 2024-05-22 sprint
Updated by Tom Clegg 3 days ago
21705-go-deps @ 03c62749d21315d1a2020ca663646cff185b63c3 -- developer-run-tests: #4217
- Migrate keepstore S3 driver, dispatchcloud EC2 driver, and keep-web S3 test case to modern AWS SDK (aws-sdk-go-v2)
- Update hashicorp/go-retryablehttp with our PR merged, remove our workaround
I have started another branch (21705-go122) with dependency updates that require go >= 1.21. Currently CI images have go 1.20, but if we rebuild after merging this branch (21705-go-deps) they will have 1.22.
Go 1.21 introduces toolchain versioning, so this sort of upgrade should get easier in future.
Updated by Tom Clegg 2 days ago
- File cloudtest.log.txt cloudtest.log.txt added
arvados-server cloudtest result looks good: cloudtest.log.txt
The "cloud/driver Create succeeded, but instance is not in list" message is logged by the previous version too:
workbench.tordo:~# /tmp/arvados-server --version /tmp/arvados-server 2.8.0~dev20240514203017 (go1.20.6) workbench.tordo:~# /tmp/arvados-server cloudtest ... ERRO[2024-05-15T14:24:31.031447498Z] cloud/driver Create succeeded, but instance is not in list error="test instance missing from cloud provider's list" ...
Evidently this is an "eventually consistent" situation that cloudtest doesn't expect. It might cause a bit of unnecessary churn in arvados-dispatch-cloud, but in the context of porting our EC2 driver to the new SDK it just indicates we have successfully preserved the existing driver's behavior.
Updated by Tom Clegg 2 days ago
21705-go-deps @ e1e4cf1e604f0d2bbf4f959123edbf0b9d3474df -- developer-run-tests: #4219
Updated by Tom Clegg 1 day ago
- If
AccessKeyIDandSecretAccessKeyare both blank, s3 driver uses IAM role - If
IAMRoleis also blank, s3 driver uses whatever IAM role is provided in instance metadata - If
IAMRoleis not blank, s3 driver uses IAM role in instance metadata only if it matches - If
AccessKeyID/SecretAccessKeyandIAMRoleare both non-blank, error out at startup because it's unclear which credentials you want to use
However, #3 and #4 were never implemented in the "s3aws" driver, which has since become the only s3 driver. The IAMRole config is just completely ignored.
For #3: I think it would be painful to implement and maintain this with the new code base. The aws sdk does lots of things under the hood so it's hard to interfere/second-guess correctly.
For #4: If we don't have #3, then #4 reduces to "set this to true to assure us that your blank access/secret keys are not an error/oversight", which seems a bit silly. I am inclined to remove theIAMRole config, so the behaviors are
- set AccessKeyID and SecretAccessKey → use explicit credentials
- don't set them → use instance metadata
Again, this is what the code has been doing since April 2023, so it would just be a documentation update.