Project

General

Profile

Bug #16392

Updated by Ward Vandewege over 4 years ago

Our dev cluster, `ce8i5`, had this in its `config.yml` file: 

 <pre> 
     Services: 
       Keepstore: 
         InternalURLs: 
           "http://10.47.0.6:25107/": {} 
 </pre> 

 Since feature #16328, keepproxy now uses those entries instead of trying to discover keepstores via the api server. 

 It turns out that the trailing slash in the InternalURLs entry causes problems. This is what happens on the keepstore in question: 

 <pre> 
 May 01 01:02:54 keep0.ce8i5.arvadosapi.com keepstore[42411]: {"PID":42411,"RequestID":"req-ck4ozsdn8womgzwr5fva","level":"info","msg":"request","remoteAddr":"10.47.0.5:39818","reqBytes":24,"reqForwardedFor":"","reqHost":"10.47.0.6:25107","reqMethod":"PUT","reqPath":"/2ad1390f7b534cf6b25765606796626a","reqQuery":"","time":"2020-05-01T01:02:54.680169064Z"} 
 May 01 01:02:54 keep0.ce8i5.arvadosapi.com keepstore[42411]: {"PID":42411,"RequestID":"req-ck4ozsdn8womgzwr5fva","level":"info","msg":"response","remoteAddr":"10.47.0.5:39818","reqBytes":24,"reqForwardedFor":"","reqHost":"10.47.0.6:25107","reqMethod":"PUT","reqPath":"/2ad1390f7b534cf6b25765606796626a","reqQuery":"","respBytes":0,"respStatus":"Moved Permanently","respStatusCode":301,"time":"2020-05-01T01:02:54.680293465Z","timeToStatus":0.000129,"timeTotal":0.000132,"timeWriteBody":0.000002} 
 May 01 01:02:54 keep0.ce8i5.arvadosapi.com keepstore[42411]: {"PID":42411,"RequestID":"req-ck4ozsdn8womgzwr5fva","level":"info","msg":"request","remoteAddr":"10.47.0.5:39818","reqBytes":0,"reqForwardedFor":"","reqHost":"10.47.0.6:25107","reqMethod":"GET","reqPath":"2ad1390f7b534cf6b25765606796626a","reqQuery":"","time":"2020-05-01T01:02:54.687455786Z"} 
 May 01 01:02:54 keep0.ce8i5.arvadosapi.com keepstore[42411]: {"PID":42411,"RequestID":"req-ck4ozsdn8womgzwr5fva","level":"info","msg":"response","remoteAddr":"10.47.0.5:39818","reqBytes":0,"reqForwardedFor":"","reqHost":"10.47.0.6:25107","reqMethod":"GET","reqPath":"2ad1390f7b534cf6b25765606796626a","reqQuery":"","respBody":"Forbidden\n","respBytes":10,"respStatus":"Forbidden","respStatusCode":403,"time":"2020-05-01T01:02:54.687560487Z","timeToStatus":0.000094,"timeTotal":0.000100,"timeWriteBody":0.000006} 
 </pre> 

 And of course keepproxy then relays the familiar, but not so informative error to the client: 

 <pre> 
 Traceback (most recent call last): 
   File "/usr/bin/arv-put", line 7, in <module> 
     main() 
   File "/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages/arvados/commands/put.py", line 1304, in main 
     writer.start(save_collection=not(args.stream or args.raw)) 
   File "/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages/arvados/commands/put.py", line 628, in start 
     self._local_collection.manifest_text() 
   File "/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages/arvados/arvfile.py", line 270, in synchronized_wrapper 
     return orig_func(self, *args, **kwargs) 
   File "/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages/arvados/collection.py", line 1014, in manifest_text 
     self._my_block_manager().commit_all() 
   File "/usr/share/python2.7/dist/python-arvados-python-client/lib/python2.7/site-packages/arvados/arvfile.py", line 816, in commit_all 
     raise KeepWriteError("Error writing some blocks", err, label="block") 
 arvados.errors.KeepWriteError: Error writing some blocks: block 2ad1390f7b534cf6b25765606796626a+24 raised KeepWriteError (failed to write 2ad1390f7b534cf6b25765606796626a after 2 attempts (wanted 2 copies but wrote 0): service https://keep.ce8i5.arvadosapi.com:443/ responded with 413 HTTP/1.1 100 Continue 
   HTTP/1.1 413 Request Entity Too Large) 
 </pre> 

 Removing the trailing slash from the InternalURLs entry resolved the problem. 

 It would probably be most user friendly if keepproxy (or our config parsing library?) can handle entries both with and without trailing slash. We've named the field `InternalURLs`, so it is to be expected that some people will put a trailing slash there, and some will not. we had this one coming.

Back