Project

General

Profile

Actions
Copy link

Bug #13305

open

keepstore starts returning 401 Unauthorized in the middle of a workflow

Added by Joshua Randall about 6 years ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Future
Story points:
-
Release:
Postponed
Release relationship:
Auto

Description

I have no idea why this suddenly started happening, but it is causing lots of problems with our production system. Example log excerpt: 

2018-03-31T22:14:11.567344145Z 2018-03-31 22:14:11 arvados.arvfile[13083] ERROR: Exception doing block prefetch
2018-03-31T22:14:11.567344145Z Traceback (most recent call last):
2018-03-31T22:14:11.567344145Z File "/usr/lib/python2.7/dist-packages/arvados/arvfile.py", line 600, in _block_prefetch_worker
2018-03-31T22:14:11.567344145Z self._keep.get(b)
2018-03-31T22:14:11.567344145Z File "/usr/lib/python2.7/dist-packages/arvados/retry.py", line 162, in num_retries_setter
2018-03-31T22:14:11.567344145Z return orig_func(self, *args, **kwargs)
2018-03-31T22:14:11.567344145Z File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 972, in get
2018-03-31T22:14:11.567344145Z return self._get_or_head(loc_s, method="GET", **kwargs)
2018-03-31T22:14:11.567344145Z File "/usr/lib/python2.7/dist-packages/arvados/keep.py", line 1099, in _get_or_head
2018-03-31T22:14:11.567344145Z "failed to read {}".format(loc_s), service_errors, label="service")
2018-03-31T22:14:11.567344145Z KeepReadError: failed to read 5b56e0116fce5730cfe21ef7d8c8ddac+868894+A2034e59c3283856f660928dceda2b4b19abc16a7@5ac00827: service http://127.0.0.1:25107/ responded with 401 HTTP/1.1 401 Unauthorized

Actions
Copy link

Also available in: Atom PDF