Arvados

I have installed Arvados as per the manual install guide however keep proxy appears to be getting in a loop.

I have 1 keepproxy and 1 Keep server as follows (using a single keep server for diagnosing this issue):

arv keep_service list

{
"kind":"arvados#keepServiceList",
"etag":"",
"self_link":"",
"offset":0,
"limit":100,
"items":[ {
"href":"/keep_services/paarv-bi6l4-o9ysyljknj2sb5i",
"kind":"arvados#keepService",
"etag":"8te2coqff7ed7s7bfns9ew4y1",
"uuid":"paarv-bi6l4-o9ysyljknj2sb5i",
"owner_uuid":"paarv-tpzed-000000000000000",
"created_at":"2019-01-12T05:25:38.036908000Z",
"modified_by_client_uuid":"paarv-ozdt8-i9wzi2vcw8jhnys",
"modified_by_user_uuid":"paarv-tpzed-1hmn7ohpbsrfqo9",
"modified_at":"2019-01-12T06:11:19.049842000Z",
"service_host":"keep.paarv.local",
"service_port":443,
"service_ssl_flag":true,
"service_type":"proxy",
"read_only":false
}, {
"href":"/keep_services/paarv-bi6l4-gdz6irkn42lslvr",
"kind":"arvados#keepService",
"etag":"c5q5itxcy1e40c39i55urz7c0",
"uuid":"paarv-bi6l4-gdz6irkn42lslvr",
"owner_uuid":"paarv-tpzed-000000000000000",
"created_at":"2019-01-12T05:25:38.533144000Z",
"modified_by_client_uuid":"paarv-ozdt8-pzhy464x5nixq26",
"modified_by_user_uuid":"paarv-tpzed-000000000000000",
"modified_at":"2019-01-12T05:25:38.533144000Z",
"service_host":"keep0.paarv.local",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk",
"read_only":false
}
],
"items_available":2
}

When I run the command:

arv put test

2019-01-12 06:45:23 arvados.arv_put²⁴²¹⁵ INFO: Resuming upload from cache file /home/vagrant/.cache/arvados/arv-put/3e8ba750817771b69b2518695ba027e3
0M / 0M 0.0% Traceback (most recent call last):
File "/usr/local/bin/arv-put", line 7, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 1218, in main
writer.start(save_collection=not(args.stream or args.raw))
File "/usr/local/lib/python2.7/dist-packages/arvados/commands/put.py", line 605, in start
self._local_collection.manifest_text()
File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 270, in synchronized_wrapper
return orig_func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1013, in manifest_text
self._my_block_manager().commit_all()
File "/usr/local/lib/python2.7/dist-packages/arvados/arvfile.py", line 815, in commit_all
raise KeepWriteError("Error writing some blocks", err, label="block")
arvados.errors.KeepWriteError: Error writing some blocks: block 2b00042f7481c7b056c4b410d28f33cf+5 raised KeepWriteError (failed to write 2b00042f7481c7b056c4b410d28f33cf (wanted 2 copies but wrote 0): service https://keep.paarv.local:443/ responded with 413 HTTP/1.1 100 Continue
HTTP/1.1 413 Request Entity Too Large)

The logs on the Keepproxy report:

{"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"request","remoteAddr":"127.0.0.1:35808","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","time":"2019-01-12T06:47:03.180421358Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Begin upload 2b00042f7481c7b056c4b410d28f33cf to https://keep.paarv.local:443","time":"2019-01-12T06:47:03.220008760Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Replicas remaining to write: 2 active uploads: 1","time":"2019-01-12T06:47:03.221620236Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"request","remoteAddr":"127.0.0.1:35814","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","time":"2019-01-12T06:47:03.233063168Z"} {"level":"info","msg":"proxy loop detected (request has Via: \"HTTP/1.1 keepproxy\"): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?","time":"2019-01-12T06:47:03.233139692Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"response","remoteAddr":"127.0.0.1:35814","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","respBytes":14,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2019-01-12T06:47:03.233165157Z","timeToStatus":0.000095,"timeTotal":0.000098,"timeWriteBody":0.000003} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Upload https://keep.paarv.local:443/2b00042f7481c7b056c4b410d28f33cf error: 500 response: loop detected","time":"2019-01-12T06:47:03.234358026Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Replicas remaining to write: 2 active uploads: 0","time":"2019-01-12T06:47:03.234390858Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Begin upload 2b00042f7481c7b056c4b410d28f33cf to https://keep.paarv.local:443","time":"2019-01-12T06:47:03.234405594Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Replicas remaining to write: 2 active uploads: 1","time":"2019-01-12T06:47:03.234418452Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"request","remoteAddr":"127.0.0.1:35816","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","time":"2019-01-12T06:47:03.236223506Z"} {"level":"info","msg":"proxy loop detected (request has Via: \"HTTP/1.1 keepproxy\"): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?","time":"2019-01-12T06:47:03.236298943Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"response","remoteAddr":"127.0.0.1:35816","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","respBytes":14,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2019-01-12T06:47:03.236329127Z","timeToStatus":0.000096,"timeTotal":0.000101,"timeWriteBody":0.000004} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Upload https://keep.paarv.local:443/2b00042f7481c7b056c4b410d28f33cf error: 500 response: loop detected","time":"2019-01-12T06:47:03.238090876Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Replicas remaining to write: 2 active uploads: 0","time":"2019-01-12T06:47:03.238123223Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Begin upload 2b00042f7481c7b056c4b410d28f33cf to https://keep.paarv.local:443","time":"2019-01-12T06:47:03.238138050Z"} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Replicas remaining to write: 2 active uploads: 1","time":"2019-01-12T06:47:03.238151012Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"request","remoteAddr":"127.0.0.1:35818","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","time":"2019-01-12T06:47:03.238364438Z"} {"level":"info","msg":"proxy loop detected (request has Via: \"HTTP/1.1 keepproxy\"): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?","time":"2019-01-12T06:47:03.238419220Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"response","remoteAddr":"127.0.0.1:35818","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","respBytes":14,"respStatus":"Internal Server Error","respStatusCode":500,"time":"2019-01-12T06:47:03.238450161Z","timeToStatus":0.000072,"timeTotal":0.000082,"timeWriteBody":0.000010} {"level":"info","msg":"DEBUG: [req-uor8xgpjp8j5dwmaq7ba] Upload https://keep.paarv.local:443/2b00042f7481c7b056c4b410d28f33cf error: 500 response: loop detected","time":"2019-01-12T06:47:03.239140910Z"} {"level":"info","msg":"127.0.0.1:35808 PUT /2b00042f7481c7b056c4b410d28f33cf 413 5 2 0 Could not write sufficient replicas: [500] loop detected","time":"2019-01-12T06:47:03.239176811Z"} {"RequestID":"req-uor8xgpjp8j5dwmaq7ba","level":"info","msg":"response","remoteAddr":"127.0.0.1:35808","reqBytes":5,"reqForwardedFor":"","reqHost":"keepproxy","reqMethod":"PUT","reqPath":"2b00042f7481c7b056c4b410d28f33cf","reqQuery":"","respBytes":57,"respStatus":"Request Entity Too Large","respStatusCode":413,"time":"2019-01-12T06:47:03.239200545Z","timeToStatus":0.058776,"timeTotal":0.058781,"timeWriteBody":0.000005}

The interesting part here is:

{"level":"info","msg":"proxy loop detected (request has Via: \"HTTP/1.1 keepproxy\"): perhaps keepproxy is misidentified by gateway config as an external client, or its keep_services record does not have service_type=proxy?","time":"2019-01-12T06:47:03.233139692Z"}

I am confused as to why it would think it is in a loop. As you can see from the output above (arv keep_service list) the services appear to be configured correctly respectively. What does this error message mean?