Bug #15045

[arvados-cloud-dispatch] commit 115cbd6482632c47fdcbbbe4abc9543e7e8e30ec breaks API host loading

Added by Ward Vandewege about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

As of 115cbd6482632c47fdcbbbe4abc9543e7e8e30ec arvados-cloud-dispatch fails with:

Mar 29 13:29:00 c97qk.arvadosapi.com systemd[1]: Starting arvados-dispatch-cloud...
Mar 29 13:29:00 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"warning","msg":"SystemRootToken missing from cluster config, falling back to ARVADOS_API_TOKEN environment variable","time":"2019-03-29T13:29:00.695994395Z"}
Mar 29 13:29:00 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"warning","msg":"Services.Controller.ExternalURL missing from cluster config, falling back to ARVADOS_API_HOST(_INSECURE) environment variables","time":"2019-03-29
Mar 29 13:29:01 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"Listen":"[::]:9006","PID":115048,"Service":"arvados-dispatch-cloud","level":"info","msg":"listening","time":"2019-03-29T13:29:01.406605587Z"}
Mar 29 13:29:01 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"error","msg":"error updating queue: parse https://{https   c97qk.arvadosapi.com   false  }/arvados/v1/api_client_authorizations/current: invalid character \"{\" i
Mar 29 13:29:01 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"info","msg":"waiting 1s before retry","time":"2019-03-29T13:29:01.407880564Z"}
Mar 29 13:29:01 c97qk.arvadosapi.com systemd[1]: Started arvados-dispatch-cloud.
Mar 29 13:29:01 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"N":0,"PID":115048,"level":"info","msg":"loaded initial instance list","time":"2019-03-29T13:29:01.573110101Z"}
Mar 29 13:29:02 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"error","msg":"error updating queue: parse https://{https   c97qk.arvadosapi.com   false  }/arvados/v1/api_client_authorizations/current: invalid character \"{\" i
Mar 29 13:29:02 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"info","msg":"waiting 1s before retry","time":"2019-03-29T13:29:02.406034481Z"}
Mar 29 13:29:03 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"error","msg":"error updating queue: parse https://{https   c97qk.arvadosapi.com   false  }/arvados/v1/api_client_authorizations/current: invalid character \"{\" i
Mar 29 13:29:03 c97qk.arvadosapi.com arvados-dispatch-cloud[115048]: {"PID":115048,"level":"info","msg":"waiting 1s before retry","time":"2019-03-29T13:29:03.406277629Z"}
...

That is with the API host and token defined as config values. When I add it to the config.yml file (new style):

Mar 29 13:53:29 c97qk.arvadosapi.com systemd[1]: Starting arvados-dispatch-cloud...
Mar 29 13:53:29 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"Listen":"[::]:9006","PID":33313,"Service":"arvados-dispatch-cloud","level":"info","msg":"listening","time":"2019-03-29T13:53:29.842080633Z"}
Mar 29 13:53:29 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"PID":33313,"level":"error","msg":"error updating queue: parse https://{https   c97qk.arvadosapi.com   false  }/arvados/v1/api_client_authorizations/current: invalid character \"{\" in 
Mar 29 13:53:29 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"PID":33313,"level":"info","msg":"waiting 1s before retry","time":"2019-03-29T13:53:29.845262273Z"}
Mar 29 13:53:29 c97qk.arvadosapi.com systemd[1]: Started arvados-dispatch-cloud.
Mar 29 13:53:29 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"N":0,"PID":33313,"level":"info","msg":"loaded initial instance list","time":"2019-03-29T13:53:29.965732628Z"}
Mar 29 13:53:30 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"PID":33313,"level":"error","msg":"error updating queue: parse https://{https   c97qk.arvadosapi.com   false  }/arvados/v1/api_client_authorizations/current: invalid character \"{\" in 
Mar 29 13:53:30 c97qk.arvadosapi.com arvados-dispatch-cloud[33313]: {"PID":33313,"level":"info","msg":"waiting 1s before retry","time":"2019-03-29T13:53:30.845396279Z"}
...

After downgrading to version 1.3.1.20190320192402-1:

Mar 29 13:54:36 c97qk.arvadosapi.com systemd[1]: Starting arvados-dispatch-cloud...
Mar 29 13:54:37 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"Listen":"[::]:9006","PID":35662,"Service":"arvados-dispatch-cloud","level":"info","msg":"listening","time":"2019-03-29T13:54:37.711254903Z"}
Mar 29 13:54:37 c97qk.arvadosapi.com systemd[1]: Started arvados-dispatch-cloud.
Mar 29 13:54:37 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"N":0,"PID":35662,"level":"info","msg":"loaded initial instance list","time":"2019-03-29T13:54:37.854359307Z"}
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-6v7kierfqpq71zi","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399697602265,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-g53p2qka5uji43t","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399577667391,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-kpcru2t7ze6bsqx","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399556037107,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-5spp83v4tsw4l1i","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399667883208,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-przvbj7f98shcl4","InstanceType":"Standard_D2_v2","PID":35662,"Priority":562948399681383698,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-wmsn71xeu1t5ndr","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399685000499,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-hba1b9dyy6nig8w","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399588516543,"State":"Queued","level":"info","msg":"adding container to queue" 
Mar 29 13:54:38 c97qk.arvadosapi.com arvados-dispatch-cloud[35662]: {"ContainerUUID":"c97qk-dz642-1pops5ei6rx8llp","InstanceType":"Standard_D1_v2","PID":35662,"Priority":562948399697616780,"State":"Queued","level":"info","msg":"adding container to queue" 
...

Related issues

Related to Arvados - Story #14807: [arvados-dispatch-cloud] Features/fixes needed before first production deployResolved01/29/2019

Associated revisions

Revision aa260bb2 (diff)
Added by Peter Amstutz about 2 months ago

Fix assigning APIHost in NewClientFromConfig refs #15045

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Ward Vandewege about 2 months ago

  • Related to Story #14807: [arvados-dispatch-cloud] Features/fixes needed before first production deploy added

#2 Updated by Ward Vandewege about 2 months ago

  • Description updated (diff)

Also available in: Atom PDF