Bug #5876: [Docker] Needs smarter dependency ordering when restarting existing containers - Arvados

Actions

Copy link

Bug #5876

closed

[Docker] Needs smarter dependency ordering when restarting existing containers

Added by Brett Smith almost 9 years ago. Updated about 5 years ago.

Status:

Closed

Priority:

Normal

Assigned To:

Category:

Docker

Target version:

Story points:

Description

I did this:

I had a functional Docker cluster going with arvdock start.
I stopped it with arvdock stop.
I rebuilt all my images with build.sh clean and build.sh.
I started the updated cluster with arvdock start.

arvdock start got stuck in an infinite loop waiting for Workbench to come up. It would never succeed, because workbench.dev.arvados was not resolvable. It seems like this happened because my SkyDNS container became newer than the others:

brinstar % docker ps -a
CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS                       PORTS               NAMES
aa55d09093dc        crosbymichael/skydns:latest    "skydns -http 0.0.0.   8 minutes ago       Exited (2) 4 minutes ago                         skydns   
33d17d174c70        crosbymichael/skydock:latest   "/go/bin/skydock -tt   8 minutes ago       Exited (0) 4 minutes ago                         skydock  
39cdecae8406        270e78ec7f90                   "/etc/apache2/foregr   24 hours ago        Exited (137) 4 minutes ago                       workbench_server
44852ae75e1f        d42b42a21b8d                   "/usr/bin/supervisor   24 hours ago        Exited (0) 4 minutes ago                         shell
76dc1099130a        8dd2456d2d15                   "/etc/apache2/foregr   24 hours ago        Exited (137) 4 minutes ago                       doc_server
33e0acd710f5        7ea84edf52e8                   "/usr/local/bin/run-   24 hours ago        Exited (0) 4 minutes ago                         keep_server_1
97f8689fe957        499746c98747                   "/usr/local/bin/run-   24 hours ago        Exited (137) 4 minutes ago                       keepproxy_server
f9cdec312113        7ea84edf52e8                   "/usr/local/bin/run-   24 hours ago        Exited (0) 4 minutes ago                         keep_server_0
f8b7cd27d4be        7ea84edf52e8                   "/usr/local/bin/run-   24 hours ago                                                         keep_data
cc31ee0ded1c        4c1f81ce07b1                   "/usr/bin/supervisor   24 hours ago        Exited (0) 4 minutes ago                         compute1
3157409a5403        4c1f81ce07b1                   "/usr/bin/supervisor   24 hours ago        Exited (0) 4 minutes ago                         compute0
d31a0abe725b        daac424050c6                   "/usr/bin/supervisor   24 hours ago        Exited (137) 4 minutes ago                       api_server

When I removed those containers, arvdock start succeeded again.

Actions

Copy link

Updated by Brett Smith almost 9 years ago

Subject changed from [Docker] Needs to be smarter about restarting existing containers to [Docker] Needs smarter dependency ordering when restarting existing containers
Target version changed from Bug Triage to Arvados Future Sprints