Bug #14507

Updated by Peter Amstutz 10 months ago

CWL jobs are failing with this:

> ApiError: <HttpError 403 when requesting https://4xphq.arvadosapi.com/arvados/v1/container_requests?alt=json returned "database connection error">

Here's the relevant log:

>

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"request","remoteAddr":"127.0.0.1:14874","reqBytes":2770,
> "reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.229777247Z"}
{"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"request","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.229777247Z"}

> Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r",
> "error":"pq:
{"RequestID":"req-i7f0fqu8xkbbf313sh9r","error":"pq: password authentication failed for user \"root\"","level":"error",
> "msg":"postgresql
\"root\"","level":"error","msg":"postgresql connect succeeded but ping failed","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198",
> "reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.413792406Z"}
failed","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.413792406Z"}

> Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller[31483]: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"response","remoteAddr":"127.0.0.1:14874", {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"response","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","respBytes":41,"respStatus":"Forbidden","respStatusCode":403,"time":"2018-11-19T14:48:18.413870566Z","timeToStatus":0.184078,"timeTotal":0.184089,"timeWriteBody":0.000012}
> "reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","respBytes":41,
> "respStatus":"Forbidden","respStatusCode":403,"time":"2018-11-19T14:48:18.413870566Z","timeToStatus":0.184078,"timeTotal":0.184089,"timeWriteBody":0.000012}

The operative error messages seem to be

@pq: password authentication failed for user \"root\"@

and

@postgresql connect succeeded but ping failed@

Perhaps connections in the db connection pool are going stale?

Back