Project

General

Profile

Actions
Copy link

Bug #20432

closed

Improve CWL runner handling 503 errors

Added by Peter Amstutz 12 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Brett Smith
Category:
CWL
Target version:
Development 2023-05-24 sprint
Story points:
-
Release:
Arvados 2.7.0
Release relationship:
Auto

Description

  1. recoverable errors, like getting container states, is not an error, and should only be logged as a warning
  2. requesting /arvados/v1/config isn't retried
  3. requesting /discovery/v1/apis/arvados/v1/rest isn't retried
    1. _thread._local object has no attribute 'api' -- throwing and handling AttributeError is intentional but maybe getattr would be better, in any event it is the API object construction that is ultimately failing
  4. requesting /arvados/v1/containers/current
  5. requesting /arvados/v1/users/current
  6. everything should use 8-10 retries
  7. FUSE command.py also calls users.current without retries -- have seen a few instances of FUSE failing to start due to 503 errors on fetching discovery doc or other endpoints required for startup

Related issues

Related to Arvados - Bug #12684: Let user specify a retry strategy on the client object, used for all API callsResolvedBrett Smith05/09/2023Actions
Actions
Copy link
 #1

Updated by Peter Amstutz 12 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Peter Amstutz 12 months ago

  • Subject changed from CWL runner error handling to Improve CWL runner handling 503 errors
Actions
Copy link
 #4

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions
Copy link
 #6

Updated by Peter Amstutz 12 months ago

  • Release set to 63
  • Status changed from In Progress to New
Actions
Copy link
 #7

Updated by Peter Amstutz 12 months ago

  • Target version changed from Future to Development 2023-05-10 sprint
Actions
Copy link
 #8

Updated by Brett Smith 12 months ago

  • Related to Bug #12684: Let user specify a retry strategy on the client object, used for all API calls added
Actions
Copy link
 #10

Updated by Peter Amstutz 12 months ago

  • Target version changed from Development 2023-05-10 sprint to Development 2023-05-24 sprint
Actions
Copy link
 #11

Updated by Peter Amstutz 12 months ago

  • Target version changed from Development 2023-05-24 sprint to Development 2023-06-07
Actions
Copy link
 #12

Updated by Peter Amstutz 11 months ago

  • Release deleted (63)
Actions
Copy link
 #13

Updated by Peter Amstutz 11 months ago

  • Target version changed from Development 2023-06-07 to To be scheduled
Actions
Copy link
 #14

Updated by Brett Smith 11 months ago

All of this is obsoleted by #12684. There should be no need to write anything separate for this anymore.

Actions
Copy link
 #15

Updated by Brett Smith 11 months ago

  • Target version changed from To be scheduled to Development 2023-05-24 sprint
  • Assigned To changed from Peter Amstutz to Brett Smith
  • Status changed from New to Resolved
Actions
Copy link
 #16

Updated by Peter Amstutz 11 months ago

Brett Smith wrote in #note-14:

All of this is obsoleted by #12684. There should be no need to write anything separate for this anymore.

Can you confirm if fetching the discovery document is also now retried by default?

Actions
Copy link
 #17

Updated by Brett Smith 11 months ago

Peter Amstutz wrote in #note-16:

Can you confirm if fetching the discovery document is also now retried by default?

It is. We now pass num_retries to googleapiclient.discovery.build, and that is used when fetching the discovery document.

Actions
Copy link
 #18

Updated by Peter Amstutz 8 months ago

  • Release set to 66
Actions
Copy link

Also available in: Atom PDF