Bug #17041
open
Upload doesn't work on arvbox-demo
Description
When trying to upload data via Workbench1, sometimes the user gets an error message: error: Forbidden -- Missing or invalid Authorization header
This was possible to reproduce using the latest arvbox-demo image (Digest sha256:3775d9a297141733b771e837f9d9f531f65a2eb02a11744a28e4daa825cc89db - https://hub.docker.com/layers/arvados/arvbox-demo/latest/images/sha256-3775d9a297141733b771e837f9d9f531f65a2eb02a11744a28e4daa825cc89db?context=explore), although wasn't happening until stopping and restarting it.
The error seem to come from keepproxy and happened with both admin and normal user accounts. The failed requests did have the auth header with a v2 token.
This was not observed while doing the same tests with the 2.1.0 tagged image.
Updated by Lucas Di Pentima over 3 years ago
Additional data:
I was able to reproduce the problem again (sometimes with the latest image, the upload works fine) and this is what I'm seeing on keepproxy logs:
2020-10-22_19:46:33.82216 time="2020-10-22T19:46:33Z" level=info msg=request RequestID=req-1kez7mrwvxr4f1amppta remoteAddr="127.0.0.1:43886" reqBytes=367977 reqForwardedFor=10.1.1.2 reqHost="10.1.1.7:25101" reqMethod=POST reqPath= reqQuery= 2020-10-22_19:46:39.83110 time="2020-10-22T19:46:39Z" level=info msg="10.1.1.2,127.0.0.1:43886: CheckAuthorizationHeader error: Head \"https://10.1.1.7:8000/arvados/v1/users/current\": x509: certificate signed by unknown authority" 2020-10-22_19:46:39.83119 time="2020-10-22T19:46:39Z" level=info msg="10.1.1.2,127.0.0.1:43886 POST / 403 367977 1 0 - Missing or invalid Authorization header"
Updated by Lucas Di Pentima over 3 years ago
Just for reference, this is the config.yml file of the failing instance:
Clusters:
x2nrs:
API: {RailsSessionSecretToken: 1rrd61lfq7zqopmua7jmtjtpu1gpfuwj5u0lwrtf8bg0rxyya1q2uu45ec4j8pv599ygatshtf0jde}
Collections: {BlobSigningKey: 1d178717wt562wpurdh8hwknxovtsszd3jq7xl8ku2ysfyf0pt9bj05o1m1utrtayynspvar2zcczw,
DefaultReplication: 1, TrustAllContent: true}
Git: {GitCommand: /usr/share/gitolite3/gitolite-shell, GitoliteHome: /var/lib/arvados-arvbox/git,
Repositories: /var/lib/arvados-arvbox/git/repositories}
Login:
Test:
Enable: true
Users:
admin: {Email: admin@example, Password: admin}
user: {Email: user@example, Password: user}
ManagementToken: vutfhfgcj47zn3ozxjjgzggqa73il9dqzp5urgu2e7f52clxk6a1a9mrqg6nk6htw1qlel60mbjnk
PostgreSQL:
Connection: {client_encoding: utf8, dbname: arvados_production, host: localhost,
password: dy3n8qxkudprk0qwznmsgpz5x, user: arvados}
ConnectionPool: 32
Services:
Composer: {ExternalURL: 'https://10.1.1.7:4200'}
Controller:
ExternalURL: https://10.1.1.7:8000
InternalURLs:
http://localhost:8003: {}
GitHTTP:
ExternalURL: https://10.1.1.7:9000/
InternalURLs:
http://localhost:9001/: {}
GitSSH: {ExternalURL: 'ssh://git@10.1.1.7:'}
Keepproxy:
ExternalURL: https://10.1.1.7:25101
InternalURLs:
http://localhost:25100: {}
Keepstore:
InternalURLs:
http://localhost:25107: {}
http://localhost:25108: {}
RailsAPI:
InternalURLs:
http://localhost:8004: {}
WebDAV:
ExternalURL: https://10.1.1.7:9002/
InternalURLs:
http://localhost:9003/: {}
WebDAVDownload:
ExternalURL: https://10.1.1.7:9004/
InternalURLs:
http://localhost:9003/: {}
WebShell:
ExternalURL: https://10.1.1.7:4202
InternalURLs: {}
Websocket:
ExternalURL: wss://10.1.1.7:8002/websocket
InternalURLs:
http://localhost:8005: {}
Workbench1: {ExternalURL: 'https://10.1.1.7:443'}
Workbench2: {ExternalURL: 'https://10.1.1.7:3001'}
SystemRootToken: 28xhbqvykvkxbtlmlmmttv1ja6vwyru68rdci5fy2bv9eq9jzfto4jjouplwjzpzv9ccfqw5t6lo3d
Users: {AutoAdminFirstUser: true, AutoSetupNewUsers: true, AutoSetupNewUsersWithRepository: true,
AutoSetupNewUsersWithVmUUID: x2nrs-2x53u-67wsncdigi7vazd, NewUsersAreActive: true}
Volumes:
x2nrs-nyw5e-000000000000000:
AccessViaHosts:
http://localhost:25107: {}
Driver: Directory
DriverParameters: {Root: /var/lib/arvados-arvbox/keep0}
x2nrs-nyw5e-111111111111111:
AccessViaHosts:
http://localhost:25108: {}
Driver: Directory
DriverParameters: {Root: /var/lib/arvados-arvbox/keep1}
Workbench: {ArvadosDocsite: 'http://10.1.1.7:8001/', SecretKeyBase: s3429l8ihi82pfxi0l2onnu98z82c1fm9mst3dmqvwxmbdmftj1tjtje5pgerk12auerdml7a6ybb}
Updated by