Arvados

Since the #21773 merge to main, our test-provision-* Jenkins jobs started to fail (for example: test-provision-debian12: #109 ). I've tried deploying a test cluster with the latest dev packages and got the issue of not being able to upload files through keep-web or keepproxy, although was able to download data just fine either through keep-web (using Workbench) or keepproxy (using arv-get from the outside)

Uploading data directly through keep (i.e.: using arv-put from the shell node) works correctly, so it seems the issue is related to server components that act as keep clients.

Also, tried uploading a file to tordo using Workbench, and got an error too.

While doing an upload through keep-web, I captured both keep-web and keepstore logs:

Keep-web¶

Sep 12 18:47:13 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-z94iriqcntir1ivaiwry","level":"info","msg":"request","remoteAddr":"10.1.1.15:48972","reqBytes":0,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"OPTIONS","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","time":"2024-09-12T18:47:13.988937205Z"}
Sep 12 18:47:13 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-z94iriqcntir1ivaiwry","level":"info","msg":"response","priority":1,"queue":"api","remoteAddr":"10.1.1.15:48972","reqBytes":0,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"OPTIONS","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","respBytes":0,"respStatus":"OK","respStatusCode":200,"time":"2024-09-12T18:47:13.989037619Z","timeToStatus":0.000095,"timeTotal":0.000095,"timeWriteBody":0.000000}
Sep 12 18:47:14 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-178lb5z57die9p8gz5ti","level":"info","msg":"request","remoteAddr":"10.1.1.15:48982","reqBytes":10718439,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"PUT","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","time":"2024-09-12T18:47:14.161948403Z"}
Sep 12 18:47:14 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-178lb5z57die9p8gz5ti","collection_file_path":"report.pdf","collection_uuid":"xarv4-4zz18-1npep8rzy4f784i","level":"info","msg":"File upload","remoteAddr":"10.1.1.15:48982","reqBytes":10718439,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"PUT","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","time":"2024-09-12T18:47:14.212947790Z","user_full_name":"Lucas Di Pentima","user_uuid":"xarv4-tpzed-3ma928fbhnnejs2"}
Sep 12 18:47:43 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-178lb5z57die9p8gz5ti","error":"sync failed: Could not write sufficient replicas: [405] method not allowed","level":"error","msg":"update() returned *errors.errorString error, changing response to HTTP 500","remoteAddr":"10.1.1.15:48982","reqBytes":10718439,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"PUT","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","time":"2024-09-12T18:47:43.191583205Z"}
Sep 12 18:47:43 workbench keep-web[15367]: {"ClusterID":"xarv4","PID":15367,"RequestID":"req-178lb5z57die9p8gz5ti","level":"info","msg":"response","priority":1,"queue":"api","remoteAddr":"10.1.1.15:48982","reqBytes":10718439,"reqForwardedFor":"123.123.123.123","reqHost":"download.xarv4.example.com","reqMethod":"PUT","reqPath":"c=xarv4-4zz18-1npep8rzy4f784i/report.pdf","reqQuery":"","respBody":"sync failed: Could not write sufficient replicas: [405] method not allowed\n","respBytes":75,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2024-09-12T18:47:43.191700379Z","timeToStatus":29.029711,"timeTotal":29.029750,"timeWriteBody":0.000038}

Keepstore¶

Sep 12 18:47:43 keep0 keepstore[7895]: {"ClusterID":"xarv4","PID":7895,"RequestID":"req-19bpla8nn24vr1b8rj5g","level":"info","msg":"request","remoteAddr":"10.1.1.15:54916","reqBytes":10718439,"reqForwardedFor":"","reqHost":"10.1.2.13:25107","reqMethod":"PUT","reqPath":"/23e12771d170009f23c89b40481718f3","reqQuery":"","time":"2024-09-12T18:47:43.190142794Z"}
Sep 12 18:47:43 keep0 keepstore[7895]: {"ClusterID":"xarv4","PID":7895,"RequestID":"req-19bpla8nn24vr1b8rj5g","level":"info","msg":"response","priority":0,"queue":"api","remoteAddr":"10.1.1.15:54916","reqBytes":10718439,"reqForwardedFor":"","reqHost":"10.1.2.13:25107","reqMethod":"PUT","reqPath":"/23e12771d170009f23c89b40481718f3","reqQuery":"","respBytes":0,"respStatus":"Moved Permanently","respStatusCode":301,"time":"2024-09-12T18:47:43.190257435Z","timeToStatus":0.000103,"timeTotal":0.000107,"timeWriteBody":0.000004}
Sep 12 18:47:43 keep0 keepstore[7895]: {"ClusterID":"xarv4","PID":7895,"RequestID":"req-19bpla8nn24vr1b8rj5g","level":"info","msg":"request","remoteAddr":"10.1.1.15:54930","reqBytes":0,"reqForwardedFor":"","reqHost":"10.1.2.13:25107","reqMethod":"GET","reqPath":"23e12771d170009f23c89b40481718f3","reqQuery":"","time":"2024-09-12T18:47:43.191265893Z"}
Sep 12 18:47:43 keep0 keepstore[7895]: {"ClusterID":"xarv4","PID":7895,"RequestID":"req-19bpla8nn24vr1b8rj5g","level":"info","msg":"response","priority":0,"queue":"api","remoteAddr":"10.1.1.15:54930","reqBytes":0,"reqForwardedFor":"","reqHost":"10.1.2.13:25107","reqMethod":"GET","reqPath":"23e12771d170009f23c89b40481718f3","reqQuery":"","respBody":"method not allowed\n","respBytes":19,"respStatus":"Method Not Allowed","respStatusCode":405,"time":"2024-09-12T18:47:43.191373814Z","timeToStatus":0.000093,"timeTotal":0.000099,"timeWriteBody":0.000006}

Notes¶

• As a first observation, I didn't found any req-id matches between keep-web and keepstore, that might be a sign of a bug, since req ids should be preserved.
• Keepstore's response to the PUT request was a 301, not sure that's correct.
• Keep-web's subsequent GET request gets a 405 (method not allowed?) response from Keepstore.

Not sure why the changes made in #21822 would cause these behaviors.