Bug #14507

Updated by Peter Amstutz 10 months ago

(update) Seems to be a configuration problem #14508

CWL jobs are failing with this:

> ApiError: <HttpError 403 when requesting https://4xphq.arvadosapi.com/arvados/v1/container_requests?alt=json returned "database connection error">

Here's the relevant log:

> Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"request","remoteAddr":"127.0.0.1:14874","reqBytes":2770,
> "reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.229777247Z"}

> Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r",
> "error":"pq: password authentication failed for user \"root\"","level":"error",
> "msg":"postgresql connect succeeded but ping failed","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198",
> "reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.413792406Z"}

> Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"response","remoteAddr":"127.0.0.1:14874",
> "reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","respBytes":41,
> "respStatus":"Forbidden","respStatusCode":403,"time":"2018-11-19T14:48:18.413870566Z","timeToStatus":0.184078,"timeTotal":0.184089,"timeWriteBody":0.000012}

The operative error messages seem to be

@pq: password authentication failed for user \"root\"@

and

@postgresql connect succeeded but ping failed@

Perhaps connections in the db connection pool are going stale?

Back