Arvados

I have having problems installing Arvados using the manual install instructions on the Arvados website.

I have 5 servers, API, Workbench, 2xKeep and SSO. I have followed the instructions exactly and everything appears to be working correctly I can create accounts/login via sso, can query api and keep etc.

When I use a legitimate username and password to login to workbench I an error on the API server logs/production.log says:

{"method":"GET","path":"/login","format":"html","controller":"UserSessionsController","action":"login","status":302,"duration":0.53,"view":0.0,"db":0.0,"location":"https://paarv.local/auth/joshid","request_id":"req-1qxq886sqrubhprra78m","client_ipaddr":"10.211.55.3","client_auth":null,"params":{"return_to":"https://workbench.paarv.local/users/welcome?return_to=%2F"},"@timestamp":"2018-11-23T00:13:19.395239898Z","@version":"1","message":"[302] GET /login (UserSessionsController#login)"}
Can't verify CSRF token authenticity {"method":"POST","path":"/arvados/v1/keep_services","format":"json","controller":"Arvados::V1::KeepServicesController","action":"create","status":200,"duration":38.18,"view":3.88,"db":14.8,"request_id":"req-17e9a2hgpj7pj1ezsv89","client_ipaddr":"127.0.0.1","client_auth":"paarv-gj3su-kyjxrmzrv77hje9","params":{"keep_service":"{ \"service_host\":\"keep0.paarv.local\", \"service_port\":25107, \"service_ssl_flag\":false, \"service_type\":\"disk\" }","_profile":"true","ensure_unique_name":"false","help":"false","keep_service_given":"true"},"@timestamp":"2018-11-23T01:06:08.936298093Z","@version":"1","message":"[200] POST /arvados/v1/keep_services (Arvados::V1::KeepServicesController#create)"}

Faraday::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/middlewares/arvados_api_token.rb:63:in `call'

There are 3 issues that stand out for me firstly the client_auth object is null, secondly the CSRF token can't be validated and lastly the SSL certificate verification error.

I am using self signed certificates which are installed on all hosts and verified by the fact that I can curl from any host to any other host using curl without needing the -k option (ignores ssl errors).

I followed the install instructions exactly and have turned all the log levels to info but I am not sure where to look next. If anybody could shed any insights onto what could be causing this it would be greatly appreciated.