Arvados

This a branch that was prepared and merged at the end of last sprint because of external time constraints. Fortunately, documentation is the easiest thing to review after-the-fact: it doesn't make commitments to outside parties the same way that code APIs do.

Please review the branch as if it were pending merge, following our normal process. No holds barred. Any issues noted should be addressed.

To help avoid duplicating too much work, I suggest you start from a base distro image, then follow the instructions to install the Arvados repository, RVM, Ruby, and Bundler. Then, on CentOS, also install the python27, postgresql92, and nginx16 Software Collections. Save the image that results, and use it as a base to follow other installation instructions for each machine role.

#6589, #6590, and #6591 are known issues.

6438-multiplatform-install-guide-TC fixes some stumbling blocks I've run into so far.

(Still more testing to do.)

Comments on 6438-multiplatform-install-guide-TC at cd6710e.

I'm not sure how I feel about including instructions for installing Software Collections directly in the documentation. Merely installing the packages is not enough; you have to enable them, or set up services to boot, at the right time to use them effectively. Because this is non-trivial, I think we want to strongly encourage users to go over and read Software Collections' own documentation. I'm worried if we give them commands to copy and paste, they're not going to do that, and then run into trouble later.

(Also, this is new information that you couldn't have known about, but we need to stop recommending the nginx16 Software Collection at all, because the Phusion Passenger documentation we point users at later includes its own copy of nginx. So don't spend any more time improving those suggestions.)

In the SSO documentation, the sentence under the "Authentication methods" header should be updated to match the improved subsection headers.

Thanks.

Brett Smith wrote:

Comments on 6438-multiplatform-install-guide-TC at cd6710e.

I'm not sure how I feel about including instructions for installing Software Collections directly in the documentation. Merely installing the packages is not enough; you have to enable them, or set up services to boot, at the right time to use them effectively. Because this is non-trivial, I think we want to strongly encourage users to go over and read Software Collections' own documentation. I'm worried if we give them commands to copy and paste, they're not going to do that, and then run into trouble later.

Good point. Reverted the direct links. I kept the "yum install" command that I added, which only works if you already went through the Software Collections stuff -- the only change from before is that you do "yum install this and that" in one copy&paste instead of two copy&paste. OK?

(Also, this is new information that you couldn't have known about, but we need to stop recommending the nginx16 Software Collection at all, because the Phusion Passenger documentation we point users at later includes its own copy of nginx. So don't spend any more time improving those suggestions.)

OK. I'll leave it at that.

In the SSO documentation, the sentence under the "Authentication methods" header should be updated to match the improved subsection headers.

Indeed. Fixed, thanks.

I've added a fix for #6589 here too, since it seemed trivial. The puma script assumed it was installed in /etc/sv/puma, which I fixed by using `pwd`. I considered using chpst -/ instead of cd rails_root but thought better of it when I considered it would make it a little bit harder for non-runit users to see what it was doing and port it to their system of choice. Hopefully the -e "$envdir" (and the screaming-env-var ARVADOS_WEBSOCKETS) is obvious enough in that respect.

Tom Clegg wrote:

Brett Smith wrote:

Comments on 6438-multiplatform-install-guide-TC at cd6710e.

I'm not sure how I feel about including instructions for installing Software Collections directly in the documentation. Merely installing the packages is not enough; you have to enable them, or set up services to boot, at the right time to use them effectively. Because this is non-trivial, I think we want to strongly encourage users to go over and read Software Collections' own documentation. I'm worried if we give them commands to copy and paste, they're not going to do that, and then run into trouble later.

Good point. Reverted the direct links. I kept the "yum install" command that I added, which only works if you already went through the Software Collections stuff -- the only change from before is that you do "yum install this and that" in one copy&paste instead of two copy&paste. OK?

Yeah, that sounds good to me. And all the changes in this branch are good, so this is good to merge. Can you note what you did and didn't get to in this work? Specifically I wonder: did you get to at least read the whole diff from #6438 for surface-level sanity? And then which components did you actually test the installation of? Thanks.

Coverage: I didn't get as far as having services running in either OS. I just started installing stuff the way it said to, and waded my way through until I ran out of time. Unfortunately this meant I got mired in Ruby and SSO installs: the "new package way" had deleted the Ruby dependencies for Debian and missed adding them for Red Hat, and the SSO recipe, well, needed help.

One related observation (although it predates the branch being reviewed) is that if you start at the top and go down the list, you click a link at the top of "installing API server" to learn how to "install ruby and bundler"; when you finish that, the "next" link tells you to install SSO provider; and now you're at the end. Should we move the Ruby page to the top (as a dependency) and put SSO with the rest of the components, instead of in a separate section down below?

The updated recipes look great in diff. I do wish we had kept the "install from source" option alongside "install from packages", rather than deleting it. But I realize that would have been a lot more work, what with keeping the document organized, accommodating different install paths, and adding prerequisite recipes for Red Hat. I don't know exactly what the right answer is here but I feel like we need some sort of "start with blank {distro,version} system, install prerequisite packages, install arvados stuff" script in our test suite -- possibly just by testing installing each built package onto a blank system of the appropriate type, relying on its encoded list of dependencies? -- and that list of prerequisites needs to be connected to the install docs somehow.

The package name "python27-python-arvados-python-client" seems ... Tourettic. But I'm sure it's very practical too.