Bug #7356

[Documentation] Various small bugs in the install guide

Added by Chen Chen almost 4 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Documentation
Target version:
Start date:
09/18/2015
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

I've encountered these bugs while install my own cluster:
  • (NA) When using self-signed SSL certs, you also need to define $GIT_SSL_NO_VERIFY.
  • (NA) When install PostgreSQL on Red Hat-based system, you need to install postgresql-server as well, and remind user to change pg_hba.conf, from ident to md5.
  • (API) When install API server, remind user that puma won't start (sv start puma will failed because of missing configuration files) before you run arvados-api-server-upgrade.sh
  • (API) When using self-signed SSL certs, you will need "sso_insecure: true" in application.yml
  • (Shell) Need "yum install libcurl-devel" before "gem install arvados-cli"
  • (Crunch) Before doing any testing ,you need to install Python SDK first in order to get access to "arv-put"
    BTW, Thanks for the kindly help from #arvados IRC channel!

Related issues

Related to Arvados - Bug #7170: [Documentation] Various small bugs in the CLI SDK install guideNew08/31/2015

Associated revisions

Revision 5a9879c9 (diff)
Added by Brett Smith almost 4 years ago

7356: CentOS 6 needs a Python 3 backport for websocket-client.

Refs #7356.

Revision 5a9879c9 (diff)
Added by Brett Smith almost 4 years ago

7356: CentOS 6 needs a Python 3 backport for websocket-client.

Refs #7356.

Revision d3171a9d
Added by Brett Smith over 3 years ago

Merge branch '7645-doc-client-max-body-size-wip'

Closes #7645, #7742. Refs #7356.

History

#1 Updated by Matthew Lesko almost 4 years ago

I just started working with Arvados and also came across some of these. I can open a separate ticket if that's easier.

For the CLI installation page - http://doc.arvados.org/sdk/cli/install.html - the only instructions are for the Ruby portion.

Checking the environment works - http://doc.arvados.org/user/getting_started/check-environment.html

but, it's a bit hidden if you ask me that the Python SDK must also be installed, even if you only want to use the command-line. Otherwise you don't get the Python utilities like 'arv-put' or 'arv-run'.

http://doc.arvados.org/sdk/python/sdk-python.html

So I'd make the http://doc.arvados.org/sdk/cli/install.html page clear that you need to 'gem install arvados-cli ; pip install arvados-python-client`.

Or if you clone the git repo, https://github.com/curoverse/arvados, then sdk/cli/bin has the proper symlinks to sdk/python/bin. (Still would need the Python & Ruby requirements of course).

#2 Updated by Chen Chen almost 4 years ago

  • (Crunch) The test will certainly fail because "internal.git" is intended to be initialize by the crunch ruby script.
  • (Compute) Prerequisites said "Packages are available for CentOS 6." but Docker only provides support for CentOS 7. This is an obvious misleading.
  • (Compute) "arvados-docker-cleaner" depends on "rh-python34-python-websocket-client", but there are no such package in SCL, either for CentOS 6 or CentOS 7. (https://www.softwarecollections.org/repos/rhscl/rh-python34/)

#3 Updated by Brett Smith almost 4 years ago

Chen Chen wrote:

  • (Crunch) The test will certainly fail because "internal.git" is intended to be initialize by the crunch ruby script.

I know we discussed this over e-mail, but to spell it out for future readers: crunch-dispatch initializes git_internal_dir if it doesn't already exist. A user following the install steps linearly won't succeed at running the test at the top of the Crunch Dispatcher install guide, because the API server will fail trying to save information to its internal repository, which doesn't exist yet.

  • (Compute) Prerequisites said "Packages are available for CentOS 6." but Docker only provides support for CentOS 7. This is an obvious misleading.

Docker packages for CentOS 6 are available through EPEL. We can provide a pointer to this just like we do for Software Collections.

This needs to be provided through the Arvados package repository. I'm working on fixing that now.

#4 Updated by Brett Smith almost 4 years ago

  • Subject changed from Minor bugs on installation document to [Documentation] Various small bugs in the install guide

#5 Updated by Brett Smith almost 4 years ago

  • Target version set to Arvados Future Sprints

#6 Updated by Brett Smith almost 4 years ago

Brett Smith wrote:

Chen Chen wrote:> > * (Compute) Prerequisites said "Packages are available for CentOS 6." but Docker only provides support for CentOS 7. This is an obvious misleading.

Docker packages for CentOS 6 are available through EPEL. We can provide a pointer to this just like we do for Software Collections.

It's probably also worth pointing out that in order for this to work, you must upgrade Linux itself through EPEL. Otherwise the cgroups support in base CentOS 6 is very old, and other Arvados components won't know how to work with it. In particular, crunchstat looks for cgroups information under /sys/fs/cgroups, but CentOS 6 has it under plain /cgroups.

It's possible that this is just a runtime mount option—but it's also possible that Docker won't work without an upgraded Linux for other reasons, too.

#7 Updated by Chen Chen almost 4 years ago

Brett Smith wrote:

Docker packages for CentOS 6 are available through EPEL. We can provide a pointer to this just like we do for Software Collections.

I found a hidden repo on dockers's website, I'm trying this out. The version is docker 1.7.1.
https://yum.dockerproject.org/repo/main/centos/6/Packages/
But the support is doubtful, as Redhat is not intended to support docker on RHEL6.
https://github.com/docker/docker/issues/14365

It's probably also worth pointing out that in order for this to work, you must upgrade Linux itself through EPEL. Otherwise the cgroups support in base CentOS 6 is very old, and other Arvados components won't know how to work with it. In particular, crunchstat looks for cgroups information under /sys/fs/cgroups, but CentOS 6 has it under plain /cgroups.
It's possible that this is just a runtime mount option—but it's also possible that Docker won't work without an upgraded Linux for other reasons, too.

EPEL won't update the kernel (as it is released via Fedora, which I fully trust), the one I wrote about is (ELRepo)[http://elrepo.org/tiki/tiki-index.php], which is quite dangerous because it is just a binary release for kernel tarball as-is.
The CentOS 6 kernel lacks kernel tree /sys/fs/cgroups, and you cannot create it. Changing /etc/cgroup.conf also failed to accomplish the goal.

#8 Updated by Chen Chen over 3 years ago

When running arvados-api-server-upgrade.sh the first time, it showed:

Ensuring directory and file permissions
chmod: cannot access ‘/var/www/arvados-api/current/tmp/cache/’: No such file or directory

the script can do a "mkdir -p" before chmod.

#9 Updated by Chen Chen over 3 years ago

  • need to add "client_max_body_size 50m;" option in nginx.conf
    The arvados.git repo is too big for a nginx default instance. Example:

root@shell:/tmp# git --git-dir arvados.git push https://git.avtst.smartquerier.com:10003/arvados.git '*:*'
Counting objects: 75490, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (18878/18878), done.
error: RPC failed; result=22, HTTP code = 413
fatal: The remote end hung up unexpectedly
Writing objects: 100% (75490/75490), 20.25 MiB | 0 bytes/s, done.
Total 75490 (delta 52437), reused 75490 (delta 52437)
fatal: The remote end hung up unexpectedly
Everything up-to-date

  • may need to add "$GIT_PATH='/usr/bin/git'" in .gitolite.rc
    I encountered the following, and playing with $PATH dones no good. Solved with additional option in .rc:

WARNING: Can't exec "git": No such file or directory at /var/lib/arvados/git/gitolite/src/lib/Gitolite/Common.pm line 146, <DATA> line 1.

#10 Updated by Chen Chen over 3 years ago

Potential Fixes are tracked on Github:
https://github.com/curoverse/arvados/pull/33
Detail logs for typos are skipped here.

Also available in: Atom PDF