Bug #14507

[controller] database ping failed

Added by Peter Amstutz 11 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-
Release:
Release relationship:
Auto

Description

(update) Seems to be a configuration problem #14508

CWL jobs are failing with this:

ApiError: <HttpError 403 when requesting https://4xphq.arvadosapi.com/arvados/v1/container_requests?alt=json returned "database connection error">

Here's the relevant log:

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller31483: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"request","remoteAddr":"127.0.0.1:14874","reqBytes":2770,
"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.229777247Z"}

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller31483: {"RequestID":"req-i7f0fqu8xkbbf313sh9r",
"error":"pq: password authentication failed for user \"root\"","level":"error",
"msg":"postgresql connect succeeded but ping failed","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198",
"reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.413792406Z"}

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller31483: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"response","remoteAddr":"127.0.0.1:14874",
"reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","respBytes":41,
"respStatus":"Forbidden","respStatusCode":403,"time":"2018-11-19T14:48:18.413870566Z","timeToStatus":0.184078,"timeTotal":0.184089,"timeWriteBody":0.000012}

The operative error messages seem to be

pq: password authentication failed for user \"root\"

and

postgresql connect succeeded but ping failed

Perhaps connections in the db connection pool are going stale?

History

#1 Updated by Peter Amstutz 11 months ago

  • Status changed from New to In Progress

#2 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#5 Updated by Peter Amstutz 11 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 11 months ago

  • Status changed from In Progress to Resolved

#7 Updated by Tom Morris 11 months ago

  • Release set to 14

Also available in: Atom PDF