Bug #14804: [keepstore] Return 5xx (not 4xx) if block is not found due to transient backend device failure - Arvados

Actions

Copy link

Bug #14804

closed

[keepstore] Return 5xx (not 4xx) if block is not found due to transient backend device failure

Added by Tom Clegg over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Lucas Di Pentima

Category:

Keep

Target version:

2019-02-27 Sprint

Start date:

02/22/2019

Due date:

% Done:

100%

Estimated time:

(Total: 0.00 h)

Story points:

Release:

Arvados v1.4 - Q1/Q2 2019

Release relationship:

Auto

Description

Currently, when keepstore is trying to read a block, if one Azure-backed volume encounters a 503 error and all other volumes return 404, keepstore returns 404 to its client. This is a non-retryable error so the client will give up.

The correct behavior is to return a 502 or 503 status in this situation.

Azure error message:

storage: service returned error: StatusCode=503, ErrorCode=ServerBusy, ErrorMessage=The server is busy.

Subtasks 1 (0 open — 1 closed)

Related issues

Actions

Copy link

Updated by Tom Morris over 3 years ago

Target version changed from Arvados Future Sprints to 2019-02-27 Sprint

Actions

Copy link

Updated by Lucas Di Pentima over 3 years ago

Assigned To set to Lucas Di Pentima

Actions

Copy link

Updated by Lucas Di Pentima over 3 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Lucas Di Pentima over 3 years ago

Updates at 601764a10 - branch 14804-keepstore-transient-backend-errors
Test run: https://ci.curoverse.com/job/developer-run-tests/1082/

When requesting a block, if keepstore gets errors from all of its volumes, the error that was being returned to the client was 404 no matter which error the volumes returned.
Now, when receiving a VolumeBusyError (transient error) from a volume backend, keepstore will return a 503 status so that the client can retry instead of mistakenly believe that the block is not there.

Actions

Copy link