[keepstore] Return 5xx (not 4xx) if block is not found due to transient backend device failure
Currently, when keepstore is trying to read a block, if one Azure-backed volume encounters a 503 error and all other volumes return 404, keepstore returns 404 to its client. This is a non-retryable error so the client will give up.
The correct behavior is to return a 502 or 503 status in this situation.
Azure error message:
storage: service returned error: StatusCode=503, ErrorCode=ServerBusy, ErrorMessage=The server is busy.
#5 Updated by Lucas Di Pentima 8 months ago
Updates at 601764a10 - branch
Test run: https://ci.curoverse.com/job/developer-run-tests/1082/
When requesting a block, if
keepstore gets errors from all of its volumes, the error that was being returned to the client was 404 no matter which error the volumes returned.
Now, when receiving a
VolumeBusyError (transient error) from a volume backend,
keepstore will return a 503 status so that the client can retry instead of mistakenly believe that the block is not there.
#6 Updated by Lucas Di Pentima 8 months ago
Re-running developer-run-tests-remainder at: https://ci.curoverse.com/job/developer-run-tests-remainder/1117/