Arvados

User reported this error. When CaptureOutput

error in CaptureOutput: Could not write sufficient replicas: [503] volume unavailable

This error is produced by keepstore when it could not write a block.

From discussion at standup 2025-01-09:

It looks like we have two keepstore clients, one is set to retry the API up to 10 times and keepclient up to 4 times, but the other client only uses the Go SDK defaults, which is ? API retries and 2 keepclient retries.

The number of retries should probably be bumped up, following the logic from here:

https://doc.arvados.org/v3.0/admin/upgrading.html#v2_6_3

Also, it would be helpful if the error message reported something about how long it spent retrying and how many attempts were made.

Also, should set LocalKeepLogsToContainerLog to "errors" (#22456) so that we can see the underlying error.