Arvados

(update) Seems to be a configuration problem #14508

CWL jobs are failing with this:

ApiError: <HttpError 403 when requesting https://4xphq.arvadosapi.com/arvados/v1/container_requests?alt=json returned "database connection error">

Here's the relevant log:

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller³¹⁴⁸³: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"request","remoteAddr":"127.0.0.1:14874","reqBytes":2770,
"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.229777247Z"}

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller³¹⁴⁸³: {"RequestID":"req-i7f0fqu8xkbbf313sh9r",
"error":"pq: password authentication failed for user \"root\"","level":"error",
"msg":"postgresql connect succeeded but ping failed","remoteAddr":"127.0.0.1:14874","reqBytes":2770,"reqForwardedFor":"144.121.5.198",
"reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","time":"2018-11-19T14:48:18.413792406Z"}

Nov 19 14:48:18 4xphq.arvadosapi.com arvados-controller³¹⁴⁸³: {"RequestID":"req-i7f0fqu8xkbbf313sh9r","level":"info","msg":"response","remoteAddr":"127.0.0.1:14874",
"reqBytes":2770,"reqForwardedFor":"144.121.5.198","reqHost":"4xphq.arvadosapi.com","reqMethod":"POST","reqPath":"arvados/v1/container_requests","reqQuery":"alt=json","respBytes":41,
"respStatus":"Forbidden","respStatusCode":403,"time":"2018-11-19T14:48:18.413870566Z","timeToStatus":0.184078,"timeTotal":0.184089,"timeWriteBody":0.000012}

The operative error messages seem to be

pq: password authentication failed for user \"root\"

and

postgresql connect succeeded but ping failed

Perhaps connections in the db connection pool are going stale?