Arvados

This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -

Is it anything related to IPv4 and IPv6 issues?

Ward Vandewege wrote:

Interesting - it looks like you either already have a skydns container running, or maybe you have another dns resolver process running that is binding to the docker ip.

You could do (as root)

netstat -anp |grep LISTEN |grep 53

and see which process it is that is bound to port 53. Does this help?

Darshan Singh wrote:

This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -

Is it anything related to IPv4 and IPv6 issues?

Hmm, doesn't look like it. Something is listening on port 53 though, on the docker container port. If you do

arvdock stop

and then

docker ps

there should be no containers running, correct?

After that

arvdock start

should proceed without error.

Thanks,
Ward.

It did not work

$ ./arvdock stop
doc_server
api_server
workbench_server
skydock
skydns
keep_server_0
keep_server_1
keepproxy_server
shell
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ ./arvdock start
start_api=false
start_compute=false
start_doc=false
start_keep=false
start_keepproxy=false
start_nameserver=false
start_sso=false
start_vm=false
start_workbench=false
Starting crosbymichael/skydns container...
skydns
/usr/bin/docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a
FATA⁰⁰⁰⁰ Error response from daemon: Cannot start container 4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a: Error starting userland proxy: listen udp 172.17.42.1:53: bind: cannot assign requested address
Starting crosbymichael/skydock container...
skydock

Ward Vandewege wrote:

Darshan Singh wrote:

This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -

Is it anything related to IPv4 and IPv6 issues?

Hmm, doesn't look like it. Something is listening on port 53 though, on the docker container port. If you do

arvdock stop

and then

docker ps

there should be no containers running, correct?

After that

arvdock start

should proceed without error.

Thanks,
Ward.

Darshan Singh wrote:

It did not work

(..)

/usr/bin/docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a
FATA⁰⁰⁰⁰ Error response from daemon: Cannot start container 4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a: Error starting userland proxy: listen udp 172.17.42.1:53: bind: cannot assign requested address

Hi Darshan,

maybe your docker installation has another network? please check it doing

sudo ip addr show
sudo ip route show

Thanks,
Nico

Darshan,

I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.

or simply do:

\curl -sSL get.arvados.org | sudo sh

Thanks,
Nico

I got past the original error.

Now I get following error when I try to login.
Web application could not be started

could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
(PG::ConnectionBad)

Nico Cesar wrote:

Darshan,

I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.

or simply do:

\curl -sSL get.arvados.org | sudo sh

Thanks,
Nico

I searched around the issues on arvados, so I ran following commands:

darshan@Linux2:~$ docker info
Containers: 12
Images: 125
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 149
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-52-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 62.85 GiB
Name: Linux2
ID: AR6V:PQ65:YCHH:X6JA:RXSC:OPNG:DIBV:OAZC:JIYZ:HSXJ:CLZG:WKNI
WARNING: No swap limit support

darshan@Linux2:~$ docker exec api_server supervisorctl status
cron EXITED May 19 02:13 PM
crunch-dispatch STARTING
munge EXITED May 19 02:13 PM
passenger RUNNING pid 7, uptime 1:00:14
postgres FATAL Exited too quickly (process log may have details)
setup RUNNING pid 9, uptime 1:00:14
setup-gitolite EXITED May 19 02:13 PM
slurm EXITED May 19 02:13 PM
ssh EXITED May 19 02:13 PM

darshan@Linux2:~$ docker exec api_server ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 0.0 61256 13536 ? Ss+ 14:13 0:08 /usr/bin/python /usr/bin/supervisord -n
root 7 0.0 0.0 175604 44336 ? Sl 14:13 0:01 ruby /usr/local/rvm/gems/ruby-2.1.5/bin/passenger start -p443 --ssl --ssl-certificate=/etc/ssl/certs/ssl-cert-snakeoil.pem --ssl-certificate-key=/etc/ssl/private/ssl-cert-snakeoil.key
root 9 0.0 0.0 20272 4232 ? S 14:13 0:00 /bin/bash /usr/local/bin/setup.sh
root 105 0.0 0.0 49936 1240 ? Ss 14:13 0:00 /usr/sbin/sshd
root 109 0.0 0.0 18884 980 ? Ss 14:13 0:00 /usr/sbin/cron
munge 133 0.0 0.0 26980 1124 ? Sl 14:13 0:00 /usr/sbin/munged
root 753 0.0 0.0 26232 2132 ? Ssl 14:13 0:00 PassengerWatchdog
root 764 0.0 0.0 33620 3132 ? Sl 14:13 0:00 PassengerLoggingAgent
root 771 0.0 0.0 32988 1000 ? Ss 14:13 0:00 PassengerWebHelper: master process /var/lib/passenger/standalone/4.0.41/webhelper-1.4.7-x86_64-linux/PassengerWebHelper -c /tmp/passenger-standalone.1dhto9j/config -p /tmp/passenger-standalone.1dhto9j/
root 772 0.5 0.0 33552 2992 ? S 14:13 0:18 PassengerWebHelper: worker process
root 774 0.0 0.0 8324 396 ? Ss 14:13 0:00 /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/TempDirToucher /tmp/passenger-standalone.1dhto9j --cleanup --daemonize --pid-file /tmp/passenger-standalone.1dhto9j/temp_dir_toucher.pid --log-file /usr/src/arvados/services/api/log/passenger.443.log
root 780 0.0 0.0 4184 580 ? S 14:13 0:00 sh -c tail -f -n 0 "/usr/src/arvados/services/api/log/production.log"
root 782 0.0 0.0 4136 580 ? S 14:13 0:00 tail -f -n 0 /usr/src/arvados/services/api/log/production.log
root 783 0.0 0.0 4184 576 ? S 14:13 0:00 sh -c tail -f -n 0 "/usr/src/arvados/services/api/log/passenger.443.log"
root 784 0.0 0.0 4136 324 ? S 14:13 0:00 tail -f -n 0 /usr/src/arvados/services/api/log/passenger.443.log
root 3354 0.6 0.0 118864 3908 ? Sl 14:47 0:10 PassengerHelperAgent
root 18994 0.0 0.0 17656 1460 ? S 15:14 0:00 /bin/bash /usr/local/bin/crunch-dispatch-run.sh
root 18997 0.0 0.0 4100 320 ? S 15:14 0:00 sleep 5
root 19093 0.0 0.0 9272 1416 ? S 15:14 0:00 /bin/bash -lc exec "$" SpawnPreparerShell /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/SpawnPreparer /usr/src/arvados/services/api SU5fUEFTU0VOR0VSADEAUFlUSE9OVU5CVUZGRVJFRAAxAE5PREVfUEFUSAAvdXNyL2xvY2FsL3J2bS9nZW1zL3J1YnktMi4xLjUvZ2Vtcy9wYXNzZW5nZXItNC4wLjQxL25vZGVfbGliAFJBSUxTX0VOVgBwcm9kdWN0aW9uAFJBQ0tfRU5WAHByb2R1Y3Rpb24AV1NHSV9FTlYAcHJvZHVjdGlvbgBOT0RFX0VOVgBwcm9kdWN0aW9uAFBBU1NFTkdFUl9BUFBfRU5WAHByb2R1Y3Rpb24ASFRUUFMAb24AU0VSVkVSX1BST1RPQ09MAEhUVFAvMS4xAFNFUlZFUl9TT0ZUV0FSRQBuZ2lueC8xLjQuNwBTQ0dJADEARE9DVU1FTlRfUk9PVAAvdXNyL3NyYy9hcnZhZG9zL3NlcnZpY2VzL2FwaS9wdWJsaWMAUVVFUllfU1RSSU5HAGZvcm1hdD0lMjJqc29uJTIyAFNFUlZFUl9OQU1FAGFwaQBSRU1PVEVfUE9SVAA1NjU2MQBSRU1PVEVfQUREUgAxMC4wLjAuMTUAU0VSVkVSX1BPUlQANDQzAFJFUVVFU1RfTUVUSE9EAEdFVABTRVJWRVJfQUREUgAxMC4wLjAuMTAAUkVRVUVTVF9VUkkAL2FydmFkb3MvdjEva2VlcF9zZXJ2aWNlcy9hY2Nlc3NpYmxlP2Zvcm1hdD0lMjJqc29uJTIyAA== /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/gems/passenger-4.0.41/helper-scripts/rack-preloader.rb
root 19096 0.0 0.0 33820 8880 ? Rl 15:14 0:00 ruby /usr/local/rvm/gems/ruby-2.1.5/bin/arv user current
root 19102 0.0 0.0 15312 1124 ? Rs 15:14 0:00 ps aux
root 19113 0.0 0.0 9272 672 ? S 15:14 0:00 /bin/bash -lc exec "$" SpawnPreparerShell /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/SpawnPreparer /usr/src/arvados/services/api SU5fUEFTU0VOR0VSADEAUFlUSE9OVU5CVUZGRVJFRAAxAE5PREVfUEFUSAAvdXNyL2xvY2FsL3J2bS9nZW1zL3J1YnktMi4xLjUvZ2Vtcy9wYXNzZW5nZXItNC4wLjQxL25vZGVfbGliAFJBSUxTX0VOVgBwcm9kdWN0aW9uAFJBQ0tfRU5WAHByb2R1Y3Rpb24AV1NHSV9FTlYAcHJvZHVjdGlvbgBOT0RFX0VOVgBwcm9kdWN0aW9uAFBBU1NFTkdFUl9BUFBfRU5WAHByb2R1Y3Rpb24ASFRUUFMAb24AU0VSVkVSX1BST1RPQ09MAEhUVFAvMS4xAFNFUlZFUl9TT0ZUV0FSRQBuZ2lueC8xLjQuNwBTQ0dJADEARE9DVU1FTlRfUk9PVAAvdXNyL3NyYy9hcnZhZG9zL3NlcnZpY2VzL2FwaS9wdWJsaWMAUVVFUllfU1RSSU5HAGZvcm1hdD0lMjJqc29uJTIyAFNFUlZFUl9OQU1FAGFwaQBSRU1PVEVfUE9SVAA1NjU2MQBSRU1PVEVfQUREUgAxMC4wLjAuMTUAU0VSVkVSX1BPUlQANDQzAFJFUVVFU1RfTUVUSE9EAEdFVABTRVJWRVJfQUREUgAxMC4wLjAuMTAAUkVRVUVTVF9VUkkAL2FydmFkb3MvdjEva2VlcF9zZXJ2aWNlcy9hY2Nlc3NpYmxlP2Zvcm1hdD0lMjJqc29uJTIyAA== /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/gems/passenger-4.0.41/helper-scripts/rack-preloader.rb
root 19114 0.0 0.0 6896 616 ? R 15:14 0:00 ps -p 19093 -o ucomm=

darshan@Linux2:~$ docker exec api_server tail -n 50 /var/log/postgresql/postgresql-9.1-main.log
2015-05-19 06:04:51 UTC LOG: database system was shut down at 2015-05-19 06:04:51 UTC
2015-05-19 06:04:51 UTC LOG: autovacuum launcher started
2015-05-19 06:04:51 UTC LOG: database system is ready to accept connections
2015-05-19 06:04:51 UTC LOG: incomplete startup packet
2015-05-19 06:04:56 UTC ERROR: must be owner of extension plpgsql
2015-05-19 06:04:56 UTC STATEMENT: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
221ae8de5a03 arvados/workbench:latest "/etc/apache2/foregr 30 minutes ago Up 30 minutes 0.0.0.0:9899->80/tcp workbench_server
65b9136e12ff arvados/shell:latest "/usr/bin/supervisor 37 minutes ago Up 37 minutes shell
607bde8a4328 arvados/doc:latest "/etc/apache2/foregr 42 minutes ago Up 42 minutes 0.0.0.0:9898->80/tcp doc_server
ada47b9fc989 arvados/keepproxy:latest "/usr/local/bin/run- 47 minutes ago Up 47 minutes 0.0.0.0:9902->9100/tcp keepproxy_server
80f7b1db5599 arvados/keep:latest "/usr/local/bin/run- 50 minutes ago Up 50 minutes 0.0.0.0:25108->25107/tcp keep_server_1
fe0bb5cdda13 arvados/keep:latest "/usr/local/bin/run- 50 minutes ago Up 50 minutes 0.0.0.0:25107->25107/tcp keep_server_0
45a329aa8a10 arvados/compute:latest "/usr/bin/supervisor 54 minutes ago Up 54 minutes compute1
2f28d4c4fe2e arvados/compute:latest "/usr/bin/supervisor 54 minutes ago Up 54 minutes compute0
2cc00e3cd849 arvados/api:latest "/usr/bin/supervisor About an hour ago Up About an hour 0.0.0.0:9900->443/tcp api_server
4446a7b38fc7 crosbymichael/skydock:latest "/go/bin/skydock tt About an hour ago Up About an hour skydock
1d8057e2624e crosbymichael/skydns:latest "skydns -http 0.0.0. About an hour ago Up About an hour 10.0.42.1:53>53/udp, 8080/tcp skydns

Darshan Singh wrote:

I got past the original error.

Now I get following error when I try to login.
Web application could not be started

could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
(PG::ConnectionBad)

Nico Cesar wrote:

Darshan,

I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.

or simply do:

\curl -sSL get.arvados.org | sudo sh

Thanks,
Nico

Thanks Darshan for the detailed information, I'll respond inline

Darshan Singh wrote:
(..)

darshan@Linux2:~$ docker exec api_server supervisorctl status
cron EXITED May 19 02:13 PM
crunch-dispatch STARTING
munge EXITED May 19 02:13 PM
passenger RUNNING pid 7, uptime 1:00:14
postgres FATAL Exited too quickly (process log may have details)
setup RUNNING pid 9, uptime 1:00:14
setup-gitolite EXITED May 19 02:13 PM
slurm EXITED May 19 02:13 PM
ssh EXITED May 19 02:13 PM

Well, that's a problem: "postgres FATAL Exited too quickly (process log may have details)"

(..)

darshan@Linux2:~$ docker exec api_server tail -n 50 /var/log/postgresql/postgresql-9.1-main.log
2015-05-19 06:04:51 UTC LOG: database system was shut down at 2015-05-19 06:04:51 UTC
2015-05-19 06:04:51 UTC LOG: autovacuum launcher started
2015-05-19 06:04:51 UTC LOG: database system is ready to accept connections
2015-05-19 06:04:51 UTC LOG: incomplete startup packet
2015-05-19 06:04:56 UTC ERROR: must be owner of extension plpgsql
2015-05-19 06:04:56 UTC STATEMENT: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';

Thanks for this last line, Unfortunately isn't telling much.

Could you do a "docker logs api_server" too? Maybe we can find a clue.

also docker exec api_server tail -n 50 /var/log/syslog

Thanks,
Nico

By The Way, try a quick

sudo ./arvdock reset
sudo ./arvdock start

sometimes tearing down the docker instances 100% clear some problems