Bug #14527

Workbench/API Authentication issue

Added by james mccallum 10 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
API
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

I have having problems installing Arvados using the manual install instructions on the Arvados website.

I have 5 servers, API, Workbench, 2xKeep and SSO. I have followed the instructions exactly and everything appears to be working correctly I can create accounts/login via sso, can query api and keep etc.

When I use a legitimate username and password to login to workbench I an error on the API server logs/production.log says:

{"method":"GET","path":"/login","format":"html","controller":"UserSessionsController","action":"login","status":302,"duration":0.53,"view":0.0,"db":0.0,"location":"https://paarv.local/auth/joshid","request_id":"req-1qxq886sqrubhprra78m","client_ipaddr":"10.211.55.3","client_auth":null,"params":{"return_to":"https://workbench.paarv.local/users/welcome?return_to=%2F"},"@timestamp":"2018-11-23T00:13:19.395239898Z","@version":"1","message":"[302] GET /login (UserSessionsController#login)"}
Can't verify CSRF token authenticity {"method":"POST","path":"/arvados/v1/keep_services","format":"json","controller":"Arvados::V1::KeepServicesController","action":"create","status":200,"duration":38.18,"view":3.88,"db":14.8,"request_id":"req-17e9a2hgpj7pj1ezsv89","client_ipaddr":"127.0.0.1","client_auth":"paarv-gj3su-kyjxrmzrv77hje9","params":{"keep_service":"{ \"service_host\":\"keep0.paarv.local\", \"service_port\":25107, \"service_ssl_flag\":false, \"service_type\":\"disk\" }","_profile":"true","ensure_unique_name":"false","help":"false","keep_service_given":"true"},"@timestamp":"2018-11-23T01:06:08.936298093Z","@version":"1","message":"[200] POST /arvados/v1/keep_services (Arvados::V1::KeepServicesController#create)"}

Faraday::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed):
app/middlewares/arvados_api_token.rb:63:in `call'

There are 3 issues that stand out for me firstly the client_auth object is null, secondly the CSRF token can't be validated and lastly the SSL certificate verification error.

I am using self signed certificates which are installed on all hosts and verified by the fact that I can curl from any host to any other host using curl without needing the -k option (ignores ssl errors).

I followed the install instructions exactly and have turned all the log levels to info but I am not sure where to look next. If anybody could shed any insights onto what could be causing this it would be greatly appreciated.


Related issues

Related to Arvados - Bug #14826: [API] Search CentOS certificates path when making callbacks to remote clustersResolved02/11/2019

History

#1 Updated by Tom Clegg 9 months ago

I expect the failing request here is API server asking SSO to verify the auth session.

Is this Centos/Fedora?

On Centos7, curl-config --ca reports /etc/pki/tls/certs/ca-bundle.crt

Workbench looks in /etc/ssl/certs/ca-certificates.crt and /etc/pki/tls/certs/ca-bundle.crt (it sounds like Workbench can make API requests, so this must be working)

API server uses Faraday defaults. It might work to set SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt (or whatever path curl-config --ca says) in your API server's environment, via systemd unit file or runit script.

https://github.com/lostisland/faraday/wiki/Setting-up-SSL-certificates

#2 Updated by james mccallum 9 months ago

Tom Clegg wrote:

I expect the failing request here is API server asking SSO to verify the auth session.

Is this Centos/Fedora?

On Centos7, curl-config --ca reports /etc/pki/tls/certs/ca-bundle.crt

Workbench looks in /etc/ssl/certs/ca-certificates.crt and /etc/pki/tls/certs/ca-bundle.crt (it sounds like Workbench can make API requests, so this must be working)

API server uses Faraday defaults. It might work to set SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt (or whatever path curl-config --ca says) in your API server's environment, via systemd unit file or runit script.

https://github.com/lostisland/faraday/wiki/Setting-up-SSL-certificates

Thanks Tom. Sorry about the delay in getting back to you, that sounds spot on to what the problem is.

Would you be able to explain a bit further what you mean by "It might work to set SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt (or whatever path curl-config --ca says) in your API server's environment, via systemd unit file or runit script."? I'm running Arvados on Ubuntu and Arvados doesn't appear to be a systemd service so I don't understand how you would add the environmental variable.

Alternatively I could possibly just move the certificates to where Faraday expects them to be. Do you know how I would find out the default location for where Arvados API expects certificates to be?

#3 Updated by Tom Morris 8 months ago

  • Related to Bug #14561: Successful workbench login redirects to the wrong endpoint added

#4 Updated by Tom Clegg 7 months ago

  • Related to deleted (Bug #14561: Successful workbench login redirects to the wrong endpoint)

#5 Updated by Ward Vandewege 7 months ago

  • Related to Bug #14826: [API] Search CentOS certificates path when making callbacks to remote clusters added

#6 Updated by Tom Morris 6 months ago

  • Target version set to To Be Groomed
  • Story points deleted (8.0)

Also available in: Atom PDF