Bug #17770

[a-d-c] while in throttle mode, the logs are full of "creating new instance" lines

Added by Ward Vandewege 7 days ago. Updated 7 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

When a-d-c gets a cloud error that indicates throttling, it uses a backoff algorithm. This works as expected, but the logs are cluttered with "creating new instance" lines, e.g.:

Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-egcge6nlcyxnhsf","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.598610198Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-aexvhfezlpd2u71","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.598616679Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-xhpx06i0ao54a3a","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.598623987Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-p8o0tdro3r7p2y2","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.598630714Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-91mie2u01zeushw","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.598637687Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-596f3cnlaxlgo7k","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705535475Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-oyclj45rigf588i","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705662410Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-30jr9xd0qpazpgb","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705698025Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-91mie2u01zeushw","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705722414Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-ka4mirn7xrx7kng","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705735563Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-sgeierm62ho9isk","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705746712Z"}
Jun 08 17:08:13 2xpu4.arvadosapi.com arvados-dispatch-cloud[22656]: {"ContainerUUID":"2xpu4-dz642-r0xkpp8ijfkunsx","InstanceType":"m58xlarge.spot","PID":22656,"level":"info","msg":"creating new instance","time":"2021-06-08T17:08:13.705758343Z"}

History

#1 Updated by Ward Vandewege 7 days ago

  • Description updated (diff)

Also available in: Atom PDF