Support #5833
closed
Cannot get Arvados installed
Added by Darshan Singh about 9 years ago. Updated almost 5 years ago.
Description
Please help me figure this out. I am very keen to use Arvados and implement many pipelines here.
$ ./arvdock start
start_api=false
start_compute=false
start_doc=false
start_keep=false
start_keepproxy=false
start_nameserver=false
start_sso=false
start_vm=false
start_workbench=false
Starting crosbymichael/skydns container...
skydns
/usr/bin/docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
327f3c5489a8c10467a3cf20c3f14dac4144f3388f0204072a2a031714b6b7a8
FATA0000 Error response from daemon: Cannot start container 327f3c5489a8c10467a3cf20c3f14dac4144f3388f0204072a2a031714b6b7a8: Error starting userland proxy: listen udp 172.17.42.1:53: bind: cannot assign requested address
Starting crosbymichael/skydock container...
skydock
/usr/bin/docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain arvados -name skydns
70417c2efeea574e108ac417ea74e5baa54be65b61c8a18b95e9d3342e209b17
Starting container: api_server
/usr/bin/docker start api_server
Started container: api_server
Starting container: compute0
/usr/bin/docker start compute0
Started container: compute0
Starting container: compute1
/usr/bin/docker start compute1
Started container: compute1
Starting container: keep_server_0
/usr/bin/docker start keep_server_0
Started container: keep_server_0
Starting container: keep_server_1
/usr/bin/docker start keep_server_1
Started container: keep_server_1
Starting container: keepproxy_server
/usr/bin/docker start keepproxy_server
Started container: keepproxy_server
Starting container: doc_server
/usr/bin/docker start doc_server
Started container: doc_server
Starting container: shell
/usr/bin/docker start shell
Started container: shell
Starting container: workbench_server
/usr/bin/docker start workbench_server
Started container: workbench_server
Waiting for Arvados to be ready.
Waiting for Arvados to be ready.
Waiting for Arvados to be ready.
My Machine
OS Ubuntu 14.04
darshan@xxx:~$ uname -a
Linux xxx 3.13.0-49-generic #83-Ubuntu SMP Fri Apr 10 20:11:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64
$ ruby -v
ruby 2.2.1p85 (2015-02-26 revision 49769) [x86_64-linux]
Files
| ticket_5833.txt (3.62 KB) ticket_5833.txt | Nico César, 05/22/2015 06:54 PM |
Updated by Ward Vandewege about 9 years ago
- Tracker changed from Bug to Support
- Assigned To set to Ward Vandewege
- Target version set to 2015-05-20 sprint
Interesting - it looks like you either already have a skydns container running, or maybe you have another dns resolver process running that is binding to the docker ip.
You could do (as root)
netstat -anp |grep LISTEN |grep 53and see which process it is that is bound to port 53. Does this help?
Updated by Darshan Singh about 9 years ago
This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -
Is it anything related to IPv4 and IPv6 issues?
Ward Vandewege wrote:
Interesting - it looks like you either already have a skydns container running, or maybe you have another dns resolver process running that is binding to the docker ip.
You could do (as root)
netstat -anp |grep LISTEN |grep 53
and see which process it is that is bound to port 53. Does this help?
Updated by Ward Vandewege about 9 years ago
- Status changed from New to In Progress
Updated by Ward Vandewege about 9 years ago
Darshan Singh wrote:
This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -Is it anything related to IPv4 and IPv6 issues?
Hmm, doesn't look like it. Something is listening on port 53 though, on the docker container port. If you do
arvdock stopand then
docker psthere should be no containers running, correct?
After that
arvdock startshould proceed without error.
Thanks,
Ward.
Updated by Darshan Singh about 9 years ago
It did not work
$ ./arvdock stop
doc_server
api_server
workbench_server
skydock
skydns
keep_server_0
keep_server_1
keepproxy_server
shell
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ ./arvdock start
start_api=false
start_compute=false
start_doc=false
start_keep=false
start_keepproxy=false
start_nameserver=false
start_sso=false
start_vm=false
start_workbench=false
Starting crosbymichael/skydns container...
skydns
/usr/bin/docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a
FATA0000 Error response from daemon: Cannot start container 4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a: Error starting userland proxy: listen udp 172.17.42.1:53: bind: cannot assign requested address
Starting crosbymichael/skydock container...
skydock
Ward Vandewege wrote:
Darshan Singh wrote:
This is what I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -Is it anything related to IPv4 and IPv6 issues?
Hmm, doesn't look like it. Something is listening on port 53 though, on the docker container port. If you do
arvdock stop
and then
docker ps
there should be no containers running, correct?
After that
arvdock start
should proceed without error.
Thanks,
Ward.
Updated by Nico César about 9 years ago
Darshan Singh wrote:
It did not work
(..)
/usr/bin/docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain arvados
4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a
FATA0000 Error response from daemon: Cannot start container 4af5e3ce34b8604f36fbf5b15a5cb29afada8461e0459a00f46ec73b4ad5b24a: Error starting userland proxy: listen udp 172.17.42.1:53: bind: cannot assign requested address
Hi Darshan,
maybe your docker installation has another network? please check it doing
sudo ip addr show
sudo ip route show
Thanks,
Nico
Updated by Nico César about 9 years ago
- Status changed from In Progress to Feedback
Updated by Nico César about 9 years ago
Darshan,
I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.
or simply do:
\curl -sSL get.arvados.org | sudo sh
Thanks,
Nico
Updated by Darshan Singh almost 9 years ago
I got past the original error.
Now I get following error when I try to login.
Web application could not be started
could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
(PG::ConnectionBad)
Nico Cesar wrote:
Darshan,
I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.
or simply do:
\curl -sSL get.arvados.org | sudo sh
Thanks,
Nico
Updated by Darshan Singh almost 9 years ago
I searched around the issues on arvados, so I ran following commands:
darshan@Linux2:~$ docker info
Containers: 12
Images: 125
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 149
Dirperm1 Supported: false
Execution Driver: native-0.2
Kernel Version: 3.13.0-52-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 62.85 GiB
Name: Linux2
ID: AR6V:PQ65:YCHH:X6JA:RXSC:OPNG:DIBV:OAZC:JIYZ:HSXJ:CLZG:WKNI
WARNING: No swap limit support
darshan@Linux2:~$ docker exec api_server supervisorctl status
cron EXITED May 19 02:13 PM
crunch-dispatch STARTING
munge EXITED May 19 02:13 PM
passenger RUNNING pid 7, uptime 1:00:14
postgres FATAL Exited too quickly (process log may have details)
setup RUNNING pid 9, uptime 1:00:14
setup-gitolite EXITED May 19 02:13 PM
slurm EXITED May 19 02:13 PM
ssh EXITED May 19 02:13 PM
darshan@Linux2:~$ docker exec api_server ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.2 0.0 61256 13536 ? Ss+ 14:13 0:08 /usr/bin/python /usr/bin/supervisord -n
root 7 0.0 0.0 175604 44336 ? Sl 14:13 0:01 ruby /usr/local/rvm/gems/ruby-2.1.5/bin/passenger start -p443 --ssl --ssl-certificate=/etc/ssl/certs/ssl-cert-snakeoil.pem --ssl-certificate-key=/etc/ssl/private/ssl-cert-snakeoil.key
root 9 0.0 0.0 20272 4232 ? S 14:13 0:00 /bin/bash /usr/local/bin/setup.sh
root 105 0.0 0.0 49936 1240 ? Ss 14:13 0:00 /usr/sbin/sshd
root 109 0.0 0.0 18884 980 ? Ss 14:13 0:00 /usr/sbin/cron
munge 133 0.0 0.0 26980 1124 ? Sl 14:13 0:00 /usr/sbin/munged
root 753 0.0 0.0 26232 2132 ? Ssl 14:13 0:00 PassengerWatchdog
root 764 0.0 0.0 33620 3132 ? Sl 14:13 0:00 PassengerLoggingAgent
root 771 0.0 0.0 32988 1000 ? Ss 14:13 0:00 PassengerWebHelper: master process /var/lib/passenger/standalone/4.0.41/webhelper-1.4.7-x86_64-linux/PassengerWebHelper -c /tmp/passenger-standalone.1dhto9j/config -p /tmp/passenger-standalone.1dhto9j/
root 772 0.5 0.0 33552 2992 ? S 14:13 0:18 PassengerWebHelper: worker process
root 774 0.0 0.0 8324 396 ? Ss 14:13 0:00 /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/TempDirToucher /tmp/passenger-standalone.1dhto9j --cleanup --daemonize --pid-file /tmp/passenger-standalone.1dhto9j/temp_dir_toucher.pid --log-file /usr/src/arvados/services/api/log/passenger.443.log
root 780 0.0 0.0 4184 580 ? S 14:13 0:00 sh -c tail -f -n 0 "/usr/src/arvados/services/api/log/production.log"
root 782 0.0 0.0 4136 580 ? S 14:13 0:00 tail -f -n 0 /usr/src/arvados/services/api/log/production.log
root 783 0.0 0.0 4184 576 ? S 14:13 0:00 sh -c tail -f -n 0 "/usr/src/arvados/services/api/log/passenger.443.log"
root 784 0.0 0.0 4136 324 ? S 14:13 0:00 tail -f -n 0 /usr/src/arvados/services/api/log/passenger.443.log
root 3354 0.6 0.0 118864 3908 ? Sl 14:47 0:10 PassengerHelperAgent
root 18994 0.0 0.0 17656 1460 ? S 15:14 0:00 /bin/bash /usr/local/bin/crunch-dispatch-run.sh
root 18997 0.0 0.0 4100 320 ? S 15:14 0:00 sleep 5
root 19093 0.0 0.0 9272 1416 ? S 15:14 0:00 /bin/bash -lc exec "$" SpawnPreparerShell /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/SpawnPreparer /usr/src/arvados/services/api SU5fUEFTU0VOR0VSADEAUFlUSE9OVU5CVUZGRVJFRAAxAE5PREVfUEFUSAAvdXNyL2xvY2FsL3J2bS9nZW1zL3J1YnktMi4xLjUvZ2Vtcy9wYXNzZW5nZXItNC4wLjQxL25vZGVfbGliAFJBSUxTX0VOVgBwcm9kdWN0aW9uAFJBQ0tfRU5WAHByb2R1Y3Rpb24AV1NHSV9FTlYAcHJvZHVjdGlvbgBOT0RFX0VOVgBwcm9kdWN0aW9uAFBBU1NFTkdFUl9BUFBfRU5WAHByb2R1Y3Rpb24ASFRUUFMAb24AU0VSVkVSX1BST1RPQ09MAEhUVFAvMS4xAFNFUlZFUl9TT0ZUV0FSRQBuZ2lueC8xLjQuNwBTQ0dJADEARE9DVU1FTlRfUk9PVAAvdXNyL3NyYy9hcnZhZG9zL3NlcnZpY2VzL2FwaS9wdWJsaWMAUVVFUllfU1RSSU5HAGZvcm1hdD0lMjJqc29uJTIyAFNFUlZFUl9OQU1FAGFwaQBSRU1PVEVfUE9SVAA1NjU2MQBSRU1PVEVfQUREUgAxMC4wLjAuMTUAU0VSVkVSX1BPUlQANDQzAFJFUVVFU1RfTUVUSE9EAEdFVABTRVJWRVJfQUREUgAxMC4wLjAuMTAAUkVRVUVTVF9VUkkAL2FydmFkb3MvdjEva2VlcF9zZXJ2aWNlcy9hY2Nlc3NpYmxlP2Zvcm1hdD0lMjJqc29uJTIyAA== /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/gems/passenger-4.0.41/helper-scripts/rack-preloader.rb" SpawnPreparerShell /var/lib/passenger/standalone/4.0.41/support-x86_64-linux/agents/SpawnPreparer /usr/src/arvados/services/api SU5fUEFTU0VOR0VSADEAUFlUSE9OVU5CVUZGRVJFRAAxAE5PREVfUEFUSAAvdXNyL2xvY2FsL3J2bS9nZW1zL3J1YnktMi4xLjUvZ2Vtcy9wYXNzZW5nZXItNC4wLjQxL25vZGVfbGliAFJBSUxTX0VOVgBwcm9kdWN0aW9uAFJBQ0tfRU5WAHByb2R1Y3Rpb24AV1NHSV9FTlYAcHJvZHVjdGlvbgBOT0RFX0VOVgBwcm9kdWN0aW9uAFBBU1NFTkdFUl9BUFBfRU5WAHByb2R1Y3Rpb24ASFRUUFMAb24AU0VSVkVSX1BST1RPQ09MAEhUVFAvMS4xAFNFUlZFUl9TT0ZUV0FSRQBuZ2lueC8xLjQuNwBTQ0dJADEARE9DVU1FTlRfUk9PVAAvdXNyL3NyYy9hcnZhZG9zL3NlcnZpY2VzL2FwaS9wdWJsaWMAUVVFUllfU1RSSU5HAGZvcm1hdD0lMjJqc29uJTIyAFNFUlZFUl9OQU1FAGFwaQBSRU1PVEVfUE9SVAA1NjU2MQBSRU1PVEVfQUREUgAxMC4wLjAuMTUAU0VSVkVSX1BPUlQANDQzAFJFUVVFU1RfTUVUSE9EAEdFVABTRVJWRVJfQUREUgAxMC4wLjAuMTAAUkVRVUVTVF9VUkkAL2FydmFkb3MvdjEva2VlcF9zZXJ2aWNlcy9hY2Nlc3NpYmxlP2Zvcm1hdD0lMjJqc29uJTIyAA== /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/wrappers/ruby /usr/local/rvm/gems/ruby-2.1.5/gems/passenger-4.0.41/helper-scripts/rack-preloader.rb
root 19096 0.0 0.0 33820 8880 ? Rl 15:14 0:00 ruby /usr/local/rvm/gems/ruby-2.1.5/bin/arv user current
root 19102 0.0 0.0 15312 1124 ? Rs 15:14 0:00 ps aux
root 19113 0.0 0.0 9272 672 ? S 15:14 0:00 /bin/bash -lc exec "$
root 19114 0.0 0.0 6896 616 ? R 15:14 0:00 ps -p 19093 -o ucomm=
darshan@Linux2:~$ docker exec api_server tail -n 50 /var/log/postgresql/postgresql-9.1-main.log
2015-05-19 06:04:51 UTC LOG: database system was shut down at 2015-05-19 06:04:51 UTC
2015-05-19 06:04:51 UTC LOG: autovacuum launcher started
2015-05-19 06:04:51 UTC LOG: database system is ready to accept connections
2015-05-19 06:04:51 UTC LOG: incomplete startup packet
2015-05-19 06:04:56 UTC ERROR: must be owner of extension plpgsql
2015-05-19 06:04:56 UTC STATEMENT: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
221ae8de5a03 arvados/workbench:latest "/etc/apache2/foregr 30 minutes ago Up 30 minutes 0.0.0.0:9899->80/tcp workbench_server
65b9136e12ff arvados/shell:latest "/usr/bin/supervisor 37 minutes ago Up 37 minutes shell
607bde8a4328 arvados/doc:latest "/etc/apache2/foregr 42 minutes ago Up 42 minutes 0.0.0.0:9898->80/tcp doc_server
ada47b9fc989 arvados/keepproxy:latest "/usr/local/bin/run- 47 minutes ago Up 47 minutes 0.0.0.0:9902->9100/tcp keepproxy_server
80f7b1db5599 arvados/keep:latest "/usr/local/bin/run- 50 minutes ago Up 50 minutes 0.0.0.0:25108->25107/tcp keep_server_1
fe0bb5cdda13 arvados/keep:latest "/usr/local/bin/run- 50 minutes ago Up 50 minutes 0.0.0.0:25107->25107/tcp keep_server_0
45a329aa8a10 arvados/compute:latest "/usr/bin/supervisor 54 minutes ago Up 54 minutes compute1
2f28d4c4fe2e arvados/compute:latest "/usr/bin/supervisor 54 minutes ago Up 54 minutes compute0
2cc00e3cd849 arvados/api:latest "/usr/bin/supervisor About an hour ago Up About an hour 0.0.0.0:9900->443/tcp api_server
4446a7b38fc7 crosbymichael/skydock:latest "/go/bin/skydock tt About an hour ago Up About an hour skydock >53/udp, 8080/tcp skydns
1d8057e2624e crosbymichael/skydns:latest "skydns -http 0.0.0. About an hour ago Up About an hour 10.0.42.1:53
Darshan Singh wrote:
I got past the original error.
Now I get following error when I try to login.
Web application could not be startedcould not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
(PG::ConnectionBad)Nico Cesar wrote:
Darshan,
I just updated arvdock so it takes in account the bridge and autodetects the IP. Please try again with the latest version.
or simply do:
\curl -sSL get.arvados.org | sudo sh
Thanks,
Nico
Updated by Ward Vandewege almost 9 years ago
- Target version changed from 2015-05-20 sprint to 2015-06-10 sprint
Updated by Nico César almost 9 years ago
Thanks Darshan for the detailed information, I'll respond inline
Darshan Singh wrote:
(..)
darshan@Linux2:~$ docker exec api_server supervisorctl status
cron EXITED May 19 02:13 PM
crunch-dispatch STARTING
munge EXITED May 19 02:13 PM
passenger RUNNING pid 7, uptime 1:00:14
postgres FATAL Exited too quickly (process log may have details)
setup RUNNING pid 9, uptime 1:00:14
setup-gitolite EXITED May 19 02:13 PM
slurm EXITED May 19 02:13 PM
ssh EXITED May 19 02:13 PM
Well, that's a problem: "postgres FATAL Exited too quickly (process log may have details)"
(..)
darshan@Linux2:~$ docker exec api_server tail -n 50 /var/log/postgresql/postgresql-9.1-main.log
2015-05-19 06:04:51 UTC LOG: database system was shut down at 2015-05-19 06:04:51 UTC
2015-05-19 06:04:51 UTC LOG: autovacuum launcher started
2015-05-19 06:04:51 UTC LOG: database system is ready to accept connections
2015-05-19 06:04:51 UTC LOG: incomplete startup packet
2015-05-19 06:04:56 UTC ERROR: must be owner of extension plpgsql
2015-05-19 06:04:56 UTC STATEMENT: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
Thanks for this last line, Unfortunately isn't telling much.
Could you do a "docker logs api_server" too? Maybe we can find a clue.
also docker exec api_server tail -n 50 /var/log/syslog
Thanks,
Nico
Updated by Nico César almost 9 years ago
By The Way, try a quick
sudo ./arvdock reset sudo ./arvdock start
sometimes tearing down the docker instances 100% clear some problems
Updated by Nico César almost 9 years ago
- File ticket_5833.txt ticket_5833.txt added
Check ticket_5833.txt
I was unable to reproduce the error in a brand new instance (this is in GCE ) running Ubuntu 14.04
do you have any extra information why postgres could be exiting promptly?
Thanks,
Nico
Updated by Brett Smith almost 9 years ago
- Target version changed from 2015-06-10 sprint to Bug Triage
Updated by Ward Vandewege almost 5 years ago
- Status changed from Feedback to Resolved