Arvados

I might be missing something, but as a new user I expected `arvados-git-httpd` and the `git` package itself to be all I needed to use git with Arvados. In other words, it seemed like the ssh key based git access would be the extra feature that could be added if desired (i.e. to integrate better with third-party tools), but that the arvados packages would be enough to get going.

I now see how the current setup offloads management of ssh-key based access to git to gitolite, and I think that makes sense if support was actually implemented that way around (i.e. first came gitolite ssh access and later you implemented arvados-git-httpd to provide https access via arvados api tokens). However, I still think it could be more straightforward if all of this could be handled by arvados-git-httpd so that gitolite was only needed if ssh access to git was desired.

Couldn't arvados-git-httpd include the functionality required to create new repos as well as accessing existing ones? If it did, couldn't the API call arvados-git-httpd to create repos syncrhonously rather than requiring a cron job? That'd probably be a feature request rather than a documentation bug, but I think it would make the process easier for new installations to get up and running with a fully functional system, on top of which ssh/gitolite could be configured if desired.

Joshua Randall wrote:

Couldn't arvados-git-httpd include the functionality required to create new repos as well as accessing existing ones? If it did, couldn't the API call arvados-git-httpd to create repos syncrhonously rather than requiring a cron job? That'd probably be a feature request rather than a documentation bug, but I think it would make the process easier for new installations to get up and running with a fully functional system, on top of which ssh/gitolite could be configured if desired.

That's right, as it happens Arvados was originally built using gitolite first and arv-git-httpd came later, so the necessary functionality (independent of gitolite) hasn't been built yet. Filed as #6769

As an interim measure/note, the gitolite setup requires having this in /var/lib/gitolite/.gitolite.rc (e.g., right at the top):

my $repo_aliases;
my $aliases_src = "$ENV{HOME}/.gitolite/arvadosaliases.pl";
if ($ENV{HOME} && (-e $aliases_src)) {
    $repo_aliases = do $aliases_src;
}
$repo_aliases ||= {};

Then, further down in the big %RC hash:

REPO_ALIASES => $repo_aliases,

And in the "ENABLE => [" section, uncomment/add:

            # access a repo by another (possibly legacy) name
            'Alias',

...like in docker/api/.gitolite.rc

The REPO_ALIASES feature is in gitolite 3.6-1 (wheezy backports) but not 2.2-1 (ubuntu1204)

http://gitolite.com/gitolite/gitolite.html#migr

• Install gitolite in ~git == /var/lib/arvados/git/. Repositories are in /var/lib/arvados/git/repositories/.
• Move docker/api/update-gitolite.rb to services/api/script/arvados-git-sync.rb.

Unsurprisingly, our gitolite sync script crashes if there's no gitolite setup to update. It's possible to simulate enough of it without actually installing gitolite, but it clutters the guide with weirdness, especially if you want it to be possible to install gitolite on top of that simulated setup without losing anything. So instead of doing that, I'm going with: arv-git-httpd depends on arvados-git-sync, which (currently) depends on gitolite.

• /var/lib/git is not ours to muck with (by imposing gitolite directory structure)
• /var/lib/gitolite and /var/lib/gitolite3 are owned by gitolite deb packages
• The repositories belong to Arvados and Arvados (sometimes) puts gitolite stuff in there. They don't belong to gitolite.

6663-bugfixes (7019bc98) has a few bugfixes that are needed to make the "set up arvados.git" part work sensibly.

I'm getting API server test failures?

  4) Failure:
Arvados::V1::CollectionsControllerTest#test_reject_manifest_with_unsigned_block_as_stream_name [/home/peter/work/arvados/services/api/test/functional/arvados/v1/collections_controller_test.rb:587]:
Expected [422, 403] to include 200.

  5) Failure:
Arvados::V1::CollectionsControllerTest#test_create_fails_with_invalid_signature [/home/peter/work/arvados/services/api/test/functional/arvados/v1/collections_controller_test.rb:553]:
Expected response to be a <403>, but was <200>

Peter Amstutz wrote:

I'm getting API server test failures?

[...]

Went away after cleaning up my application.yml, so disregard.

Comments on 6663-bugfixes:

• In RepositoriesController#get_all_permissions it looks like if a user is a member of more than one group which has a permission link to a repository, it will emit multiple entries into the "perms" list. Later on it uses the entries in "perms" to set can_manage/can_write/can_read flags, and then emits the final permissions based on precedence, but the code pretty dense so I'm not 100% confident that it will do the right thing in every case. Please add comments / undensify.

• This is aliasing the previous perms variable which is confusing:
  

  repo_users[:user_permissions].each do |user_uuid,perms|
  

• The "name" key is alternately used to mean both the repository name and the permission name.

Indeed. Updated with explanatory comments, and fixed a few vague/misleading variable names (which were already my fault before this branch).

The "name" key is alternately used to mean both the repository name and the permission name.

I think this is OK because [a] it follows the database schema, [b] the only place "name" is used to refer to a repository name is obviously a repo name ("name: repo.name,") and is visually well separated from the other uses. Is this fair?

Now at 8699ba0

Came across a confusing bug/limitation that should probably be documented, and logged better by arv-git-httpd when it comes up.

Go's CGI library doesn't accept chunked encoding: http://golang.org/src/net/http/cgi/host.go?s=3019:3216#L97

But git uses chunked encoding during push, so you get this.

$ git --git-dir /tmp/arvados.git push http://bogus.arvadosapi.com:9001/arvados.git '*:*'
Counting objects: 70450, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (17742/17742), done.
error: RPC failed; result=22, HTTP code = 400
fatal: The remote end hung up unexpectedly
Writing objects: 100% (70450/70450), 19.29 MiB | 0 bytes/s, done.
Total 70450 (delta 48690), reused 70450 (delta 48690)
fatal: The remote end hung up unexpectedly
Everything up-to-date

2015-07-31_06:05:49.95843 2015/07/31 06:05:49 "127.0.0.1:39955" "none" "22mscckxkil6rin1h5lxb0a1v5224nzm3ysc5g5l8l5u9v7wf3" 200 "read" "arvados" "GET" "/bogus-s0uqq-p6ay7mxx32ixcjd.git/info/refs" 
2015-07-31_06:05:50.46958 2015/07/31 06:05:50 "127.0.0.1:39957" "none" "22mscckxkil6rin1h5lxb0a1v5224nzm3ysc5g5l8l5u9v7wf3" 200 "write" "arvados" "POST" "/bogus-s0uqq-p6ay7mxx32ixcjd.git/git-receive-pack" 
2015-07-31_06:05:50.56426 2015/07/31 06:05:50 "127.0.0.1:39957" "none" "22mscckxkil6rin1h5lxb0a1v5224nzm3ysc5g5l8l5u9v7wf3" 400 "write" "arvados" "POST" "/bogus-s0uqq-p6ay7mxx32ixcjd.git/git-receive-pack" 

• Default: proxy_http_version 1.0; (if this is what we're relying on, the nginx.conf in the docs should configure it explicitly.)
• Similar to http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html -> "proxy_sendcl".

At minimum, arv-git-httpd should detect chunked encoding before handing off to the Go library, and logging something more helpful than just "400", perhaps "A proxy is required to serve this client" with a link to a doc page with more details.

But ideally, we can explain to git -- or whatever proxy is immediately upstream of arv-git-httpd -- that it shouldn't send requests with chunked encoding in the first place. This might be as easy as always responding with HTTP/1.0 in arv-git-httpd? (The basic issue here seems to be that HTTP/1.1 and CGI are mutually incompatible: chunked encoding is always acceptable in HTTP/1.1 and never acceptable in CGI.)

6663-git-server preview at http://6663.tom.shell.9tee4.arvadosapi.com/install/install-arv-git-httpd.html

So, this is awesome, all our documentation should be like this.

• Example secret keys should be obviously bogus ("zzzzzzzzzzzzzzzzzzzzz") so that people don't cut and paste the examples into their files resulting in easily Googleable security holes.
• It would be nice to have a section on how to test if the installation works (e.g. ssh to git@... to access gitolite)
• There's nothing about setting up the initial system "arvados" repository. I think from talking with Josh we established that having those instructions in the "Install API" instructions was wrong, because it will look for the arvados repository by uuid and not as arvados.git.

I'm running out of time and was not able to get to arvados-git-httpd.

Also

git@164d402d3965:~/gitolite-admin$ (cd /var/www/arvados-api/current && /usr/local/rvm/bin/rvm-exec default bundle exec script/arvados-git-sync.rb production)
error: cannot open .git/FETCH_HEAD: Permission denied

Peter Amstutz wrote:

Also

[...]

I think maybe I ran the script as root and so FETCH_HEAD was owned by root instead of git.

Related to instructions for the user to test things out: the user needs to create at least one repository first, otherwise ssh git@... will ask for a password.

Peter Amstutz wrote:

So, this is awesome, all our documentation should be like this.

Thanks! Funny, while writing it I was thinking "this is too much, it should be automated". But I do really like the aspect of saying enough about what's happening that it's possible to debug when things don't go exactly as planned. Hopefully this stuff will still be useful in that way when we do automate more. Anyway, I'm glad it wasn't as hard to read as it was to write. :)

• Example secret keys should be obviously bogus ("zzzzzzzzzzzzzzzzzzzzz") so that people don't cut and paste the examples into their files resulting in easily Googleable security holes.

Fixed, 97fa1e2. (In this particular case copying and pasting wouldn't actually cause the example secret to be accepted by anything, it would just fail to update the repos -- but yes, for the sake of clarity and habit-forming zzzzzz is definitely better.)

• It would be nice to have a section on how to test if the installation works (e.g. ssh to git@... to access gitolite)

True... "ssh git@localhost" tests gitolite itself, and "push to arvados.git" (see below) tests almost everything else, but it's possible SSH keys aren't getting installed correctly by the update script. There should probably be an "add your SSH key, create a repo, and push to it to make sure that all works" item on the (currently nonexistent?) "things to test" checklist.

OK to punt this?

• There's nothing about setting up the initial system "arvados" repository. I think from talking with Josh we established that having those instructions in the "Install API" instructions was wrong, because it will look for the arvados repository by uuid and not as arvados.git.

I moved this into install/create-standard-objects.html, after the "arv repository create" bit: doing it later from a shell node means it gets to confirm the git config setup on the shell node as well as the gitolite setup. I reduced the step on the API install page to creating an empty repositories dir, so the API config check doesn't fail ENOENT.

I'm running out of time and was not able to get to arvados-git-httpd.

Yeah, I know the feeling! See automation comment above. :)

Peter Amstutz wrote:

Related to instructions for the user to test things out: the user needs to create at least one repository first, otherwise ssh git@... will ask for a password.

If I'm following, this sounds like a bug in repositories/get_all_permissions and/or arvados-git-update.rb: "User with no repos should still have key added to gitolite config." Possibly not noticed before because we usually set up the public "arvados" repo?

• http://6663.tom.shell.9tee4.arvadosapi.com/install/install-arv-git-httpd.html
• http://6663.tom.shell.9tee4.arvadosapi.com/install/create-standard-objects.html

On `install-arv-git-httpd.html`, "DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches your git server." is a little bit unclear.

Perhaps it could be something like: "DNS and network configuration should be set up so port 443 reaches the HTTPS proxy (e.g. nginx) behind which arvados-git-httpd is running (i.e. such that a request to https://git.uuid_prefix.your.domain/ reaches arvados-git-httpd), and port 22 reaches the ssh daemon with gitolite configured for the 'git' user."

There is a typo: "The git user needs its own SSH key. (It must be able to run ssh gitlocalhost@ from scripts.)"

Should be: "The git user needs its own SSH key. (It must be able to run `ssh git@localhost` from scripts.)"

Comments on 7e77373. I have not had the opportunity to test your instructions yet; but I wanted to provide some initial feedback to keep things moving as much as possible.

• The story says, "Link to the gitolite installation documentation for details." In general, I feel like we should link to third-party documentation whenever appropriate. It's faster for us; it frees us from having to worry about version changes in that software; and it puts resources in front of the user if they run into trouble during this process.
  I would still like to at least see that link provided. I'm going back and forth with myself on whether we should forego the "clone and install Gitolite" instructions in favor of the link. I'm persuadable.
• It looks like this branch changes the default value for git_repositories_dir in application.default.yml, but most (all?) of the documentation is still written with the expectation that it's set to /var/lib/arvados/git. Is this maybe supposed to change to /var/lib/arvados/git/repositories? Or is something else going on here?
• In the "Create standard objects" page, the directory listed for the git push line is /tmp/arvados, but that directory doesn't seem to exist from following the instructions, and there's no instruction to cd into it. I think this should just be /tmp?
• The API server installation doc refers to the git installation page as the "next page." Please avoid this since we may shuffle pages around (and #6851 makes that a real likelihood).
• Why remove the sentence about the default value of git_repositories_dir? It sounds like it should be updated, but documenting the default still seems helpful.
• The Git server installation doc says, "DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches your git server." I worry that referring to "your git server" here could be confusing, since we're referring to the entire system as the git server throughout this page, and Gitolite doesn't run any daemon software besides. Suggest "your SSH server" instead.
• "Hosted repositories will be stored under /var/lib/arvados/git/. If you choose a different location, make sure to update the git_repositories_dir entry in your API server's config/application.yml file, preserving the trailing repositories/ part." - I worry about how this reads, because the first sentence makes it sound like you have no choice, while the second does; and the "trailing repositories/ part" text seems difficult to follow without a clear reference on this page.
  What if we explained this along the lines of: We're going to set up the user account and home directory for Gitolite. The default home directory is /var/lib/arvados/git, so these instructions will set it up there. You may choose a different location, but if you do, you must update the API server's git_repositories_dir setting so it refers to the repositories/ subdirectory of this home directory.
• The monospace text here causes a Textile markup hiccup: "The git user needs its own SSH key. (It must be able to run @ssh git@localhost@ from scripts.)"
• Please make the link to the Gitolite repository a real link.
• "The API server package includes a script that… creates new empty repositories if needed." This last point is not strictly correct. The script creates no repositories. It only writes Gitolite configuration; and then Gitolite creates the repositories. I think being clear on this point can help readers understand why Gitolite is a dependency, and debug any issues that might arise.
• The runit installation instructions include sudo apt-get install runit. If necessary, we should split that part out for Debian- and Red Hat-based distributions.
• For the nginx configuration, I suggest following the pattern used in the install guide for our Rails servers.
• The page references an API configuration setting git_http_base. However, the API server code doesn't seem to respect this variable. Should this be git_repo_https_base instead?

I think you should leave the specific instructions on how to install gitolite (but also link to the gitolite docs). Grokking the gitolite docs was not all that straightforward and as it turns out, Arvados needs a relatively new version to work (perhaps the need for v3 should be explicitly mentioned?).

Brett Smith wrote:

• The story says, "Link to the gitolite installation documentation for details." In general, I feel like we should link to third-party documentation whenever appropriate. It's faster for us; it frees us from having to worry about version changes in that software; and it puts resources in front of the user if they run into trouble during this process.
  I would still like to at least see that link provided. I'm going back and forth with myself on whether we should forego the "clone and install Gitolite" instructions in favor of the link. I'm persuadable.

OK, then here goes: I think this guide should tell you exactly what to do, and avoid sending you off on "figure out which parts of their suits-every-imaginable-purpose install guide you're supposed to follow" adventures. Also, I don't think "follow their docs" frees us from worrying about version changes at all: if you're worried about whether our install recipe works with a new version, you should probably be more concerned about whether our usage still works. That's why bundler exists, why our instructions use a known-good version of Ruby, etc., right?

Instead of that, how about changing the "pick a version" stuff so it says "use 3.6.x" or even "3.6.3", as we would (conceptually) if this were installed by bundler or if we were building a debian package, rather than saying "pick a stable version and then follow our 3.6.3 instructions"? For now I've changed "e.g., v3.6.3" to "this guide was tested with v3.6.3".

Joshua Randall wrote:

(perhaps the need for v3 should be explicitly mentioned?).

...and added a note specifically that <3.0 definitely won't work.

I've also added a link to the gitolite home page, but here's a vote to move it into a (new) section of troubleshooting resources: "if installing thing X didn't work, here's a good place to go for help." Similar things can happen for RVM, Postgres, nginx, "apt-get install" in general, etc., but I think the general strategy should be that our instructions suffice to get everything working.

Brett Smith wrote:

• It looks like this branch changes the default value for git_repositories_dir in application.default.yml, but most (all?) of the documentation is still written with the expectation that it's set to /var/lib/arvados/git. Is this maybe supposed to change to /var/lib/arvados/git/repositories? Or is something else going on here?

Ah, apparently I didn't go back and fix this after I changed my mind about what the new default should be. Fixed.

• In the "Create standard objects" page, the directory listed for the git push line is /tmp/arvados, but that directory doesn't seem to exist from following the instructions, and there's no instruction to cd into it. I think this should just be /tmp?

Yes, fixed.

• The API server installation doc refers to the git installation page as the "next page." Please avoid this since we may shuffle pages around (and #6851 makes that a real likelihood).

Fixed, now: The git server setup process is covered on "its own page":[...]

• Why remove the sentence about the default value of git_repositories_dir? It sounds like it should be updated, but documenting the default still seems helpful.

Generally I don't especially like documenting defaults here because it can go out of sync with the real defaults, but in this case it has to stay in sync anyway because the install guide has to tell you to mkdir the right path. I was mainly just trying to avoid saying it three times. Updated, see ed2e9fe

Honestly I think it would make more sense to delete this whole section and have "arvados-api-server-upgrade.sh" script create that directory by itself. AFAIK this section exists not because "default is likely to be wrong for you", but because "we had to put the mkdir command somewhere".

• The Git server installation doc says, "DNS and network configuration should be set up so port 443 reaches your HTTPS proxy, and port 22 reaches your git server." I worry that referring to "your git server" here could be confusing, since we're referring to the entire system as the git server throughout this page, and Gitolite doesn't run any daemon software besides. Suggest "your SSH server" instead.

Changed to "the OpenSSH service on your git server" - OK? (I defined "this box is your git server" at the top so "SSH server" alone seems like it could refer to a different box. Although AFAIK we haven't documented how to set up a switchyard yet.)

• "Hosted repositories will be stored under /var/lib/arvados/git/. If you choose a different location, make sure to update the git_repositories_dir entry in your API server's config/application.yml file, preserving the trailing repositories/ part." - I worry about how this reads, because the first sentence makes it sound like you have no choice, while the second does; and the "trailing repositories/ part" text seems difficult to follow without a clear reference on this page.

Yes, that is a bit confusing. Reworded to:

Gitolite and some additional scripts will be installed in /var/lib/arvados/git, which means the repository data will be stored in /var/lib/arvados/git/repositories. If you choose to install gitolite in a different location, make sure to update the git_repositories_dir entry in your API server's config/application.yml file accordingly: for example, if you install gitolite at /data/gitolite then your git_repositories_dir will be /data/gitolite/repositories.

• The monospace text here causes a Textile markup hiccup: "The git user needs its own SSH key. (It must be able to run @ssh git@localhost@ from scripts.)"

Fixed.

• Please make the link to the Gitolite repository a real link.

Whoops. Expected too much from textile, didn't proofread. Fixed.

• "The API server package includes a script that… creates new empty repositories if needed." This last point is not strictly correct. The script creates no repositories. It only writes Gitolite configuration; and then Gitolite creates the repositories. I think being clear on this point can help readers understand why Gitolite is a dependency, and debug any issues that might arise.

Fixed.

• The runit installation instructions include sudo apt-get install runit. If necessary, we should split that part out for Debian- and Red Hat-based distributions.

Hm. Apparently you can't yum install runit. I've left it at "install from source[link] or use an alternative daemon supervisor", but both options seem like cop-outs. Suggestions?

• For the nginx configuration, I suggest following the pattern used in the install guide for our Rails servers.

Updated to use similar nginx options and formatting. Is that what you meant?

• The page references an API configuration setting git_http_base. However, the API server code doesn't seem to respect this variable. Should this be git_repo_https_base instead?

Ah, yes. And the value needed https:// on it. And a trailing slash.

Added a corresponding section for the SSH side, too.

Added "restart nginx" to the end of the git page, and added the specific command ("sudo nginx -s reload") to git and API pages.

http://6663.tom.shell.9tee4.arvadosapi.com/install/install-arv-git-httpd.html now at 01a76d3

Tom Clegg wrote:

Brett Smith wrote:

• The story says, "Link to the gitolite installation documentation for details." In general, I feel like we should link to third-party documentation whenever appropriate. It's faster for us; it frees us from having to worry about version changes in that software; and it puts resources in front of the user if they run into trouble during this process.
  I would still like to at least see that link provided. I'm going back and forth with myself on whether we should forego the "clone and install Gitolite" instructions in favor of the link. I'm persuadable.

OK, then here goes: I think this guide should tell you exactly what to do, and avoid sending you off on "figure out which parts of their suits-every-imaginable-purpose install guide you're supposed to follow" adventures. Also, I don't think "follow their docs" frees us from worrying about version changes at all: if you're worried about whether our install recipe works with a new version, you should probably be more concerned about whether our usage still works. That's why bundler exists, why our instructions use a known-good version of Ruby, etc., right?

Instead of that, how about changing the "pick a version" stuff so it says "use 3.6.x" or even "3.6.3", as we would (conceptually) if this were installed by bundler or if we were building a debian package, rather than saying "pick a stable version and then follow our 3.6.3 instructions"? For now I've changed "e.g., v3.6.3" to "this guide was tested with v3.6.3".

Okay, so, to be clear up front, what you have here is fine. I'm not asking for any changes.

To continue the philosophical discussion: I'm all for providing our own installation instructions when we can provide steps that we're confident that will work across the board, but I worry that's often not feasible with major third-party tools like Gitolite. There are just too many variations across distributions and setups to worry about. Those variations are much more likely to be handled well by third-party documentation than our own. I mean, it seems to me this is basically the situation we're in with runit right now.

I'm also +1 to offering more detailed version advice (we've tested with this version; we know these versions won't work), but I don't think we should tell people that they must use one specific version unless we know for sure that's the only version that works. That kind of version management is fine when you have tools like Bundler to deal with it, but when that's not the case (which seems like is always true for standalone third-party software), I would hate for would-be installers to get snagged in a situation where vA.B.C is buggy on their exact distro, or whatever, and they're worried they can't use the next minor version up or down.

Honestly I think it would make more sense to delete this whole section and have "arvados-api-server-upgrade.sh" script create that directory by itself. AFAIK this section exists not because "default is likely to be wrong for you", but because "we had to put the mkdir command somewhere".

I think I'm fine with this idea, but let's not do it in this branch.

• The runit installation instructions include sudo apt-get install runit. If necessary, we should split that part out for Debian- and Red Hat-based distributions.

Hm. Apparently you can't yum install runit. I've left it at "install from source[link] or use an alternative daemon supervisor", but both options seem like cop-outs. Suggestions?

Per what I wrote above: no. :)

• For the nginx configuration, I suggest following the pattern used in the install guide for our Rails servers.

Updated to use similar nginx options and formatting. Is that what you meant?

I had in mind taking the entire <ol> and porting it forward. I realize that someone following our instructions normally should already have nginx installed, so if we want to avoid it for that reason, fine. But from here:

• Please specifically mention that the suggested configuration should go in the http section, as on other pages.
• I'm skeptical about the suggested proxy headers. I know for sure X-External-Client isn't appropriate, because that's specific to the API server, and requires additional nginx configuration. We're currently not setting any headers in production, so I think we can do without them here as well.

Additional comments from 01a76d3:

• On the API server installation page, "If you intend to store your git repositories are stored in a different location…" needs some words removed.
• On the git server installation page, we should avoid references to config/application.yml, and prefer just application.yml. This is because our packages make /etc/arvados/api/application.yml the canonical copy, so including config/ in the path seems liable to cause confusion.

Thanks.

As an end user, I find myself somewhat annoyed with the repeated suggestions to use runit to supervise processes. I think I'd prefer that you tell me what daemons you want to be running (i.e. run this command as this user) and leave it as an exercise to the reader to arrange for them to be supervised (or not).

I suppose this may not seem fair of me because I was just making the opposite point in terms of gitolite, but I actually think it is reasonable because there is a big difference between gitolite and runit - namely, that gitolite is an actual dependency of the arvados system whereas runit is just one of many possible supervision systems that could monitor the arvados daemons. It makes sense to provide specific setup instructions including a known-working version of gitolite, but I think it doesn't make sense to provide specific instructions for anything that isn't actually a dependency. If you need a deamon to be run, tell us how to run the daemon from the command line and suggest to us that in production we should arrange to supervise the process. One can have a working system with all of the daemons running in a tmux or screen session (with ones fingers crossed), and I think that is a sensible step to go through when starting out.

I guess I have a few reasons for this:
1. when initially trying to get something to work, I would prefer to just run the command interactively so I can see if there are any errors before jumping right in to having it supervised. if you recommend exactly what command to run from the command line including whatever user and environment needs to be put in place, then it is trivial to do this initial debug running, and I am then in a good position to setup supervision using whatever system I prefer.

2. if we are already using another system (like supervisord) to manage processes and prefer to keep doing that (e.g. because we already have monitoring dashboards setup for it) and want to continue using that, we now have to learn enough of the runit syntax in order to figure out how to transcode them into supervisord (rather than you just telling us what is actually required without encoding it in runit files).

3. there seems to be some heterogeneity in runit versions and support of some features, and you could avoid those issues by not discussing it.

Brett Smith wrote:

To continue the philosophical discussion: I'm all for providing our own installation instructions when we can provide steps that we're confident that will work across the board, but I worry that's often not feasible with major third-party tools like Gitolite. There are just too many variations across distributions and setups to worry about. Those variations are much more likely to be handled well by third-party documentation than our own. I mean, it seems to me this is basically the situation we're in with runit right now.

Shrug. I suppose the effectiveness of the "install gitolite" recipe could vary appreciably more than the outcome of "apt-get install libwhatever". But at least it doesn't come with the unlimited possibilities of vendor-introduced "improvements".

Either way, I'm more inclined to be more explicit about what's expected, rather than pretend our install docs have no prerequisites/assumptions at all, and whack-a-mole the actual prerequisites as we encounter reality. "If you're installing on one of these supported distros, these steps are known to work. Otherwise, here are some helpful tips, have fun." IOW installing on FreeBSD can involve going off to follow generalized install guides at third-party doc sites and cross-reference them with your understanding of what Arvados needs from component X. Installing on jessie should not.

Honestly I think it would make more sense to delete this whole section and have "arvados-api-server-upgrade.sh" script create that directory by itself. AFAIK this section exists not because "default is likely to be wrong for you", but because "we had to put the mkdir command somewhere".

I think I'm fine with this idea, but let's not do it in this branch.

Agreed, just forgot to say that explicitly. :)

I had in mind taking the entire <ol> and porting it forward. I realize that someone following our instructions normally should already have nginx installed

Yes, I figured it was appropriate to make that assumption because it's stated explicitly that you shouldn't bother installing arvados-git-httpd anywhere but on your API server, and the API server page told you to set up Nginx.

• Please specifically mention that the suggested configuration should go in the http section, as on other pages.

Fixed.

• I'm skeptical about the suggested proxy headers. I know for sure X-External-Client isn't appropriate, because that's specific to the API server, and requires additional nginx configuration. We're currently not setting any headers in production, so I think we can do without them here as well.

Indeed, removed the extra headers and options that aren't relevant.

• On the API server installation page, "If you intend to store your git repositories are stored in a different location…" needs some words removed.

Fixed.

• On the git server installation page, we should avoid references to config/application.yml, and prefer just application.yml. This is because our packages make /etc/arvados/api/application.yml the canonical copy, so including config/ in the path seems liable to cause confusion.

Fixed.

Joshua Randall wrote:

As an end user, I find myself somewhat annoyed with the repeated suggestions to use runit to supervise processes. I think I'd prefer that you tell me what daemons you want to be running (i.e. run this command as this user) and leave it as an exercise to the reader to arrange for them to be supervised (or not).

The current plan here is to add startup scripts (sysv, runit, upstart) to every service package. I expect/hope this will involve a script shared by all three systems which can also be used from a terminal window.

I suppose this may not seem fair of me because I was just making the opposite point in terms of gitolite, but I actually think it is reasonable because there is a big difference between gitolite and runit - namely, that gitolite is an actual dependency of the arvados system whereas runit is just one of many possible supervision systems that could monitor the arvados daemons. It makes sense to provide specific setup instructions including a known-working version of gitolite, but I think it doesn't make sense to provide specific instructions for anything that isn't actually a dependency. If you need a deamon to be run, tell us how to run the daemon from the command line and suggest to us that in production we should arrange to supervise the process. One can have a working system with all of the daemons running in a tmux or screen session (with ones fingers crossed), and I think that is a sensible step to go through when starting out.

I think I'd restate this as "runit is a dependency; it shouldn't be a dependency; and it looks relatively easy to remove our dependency on it."

It used to seem reasonable to bring runit in this way because you can install it (and use it to run things x,y,z, just use it for logging or chpst, or not use it at all) without affecting the way the rest of your system works. Unfortunately it no longer seems quite so reasonable because [a] the post-install script (or something) in the debian package kills your docker container during apt-get install (probably trying to SIGHUP pid 1 when it's bash instead of init), although it's unintrusive after that point; [b] it's not trivial to install on a Red Hat system. Oh well.

I guess I have a few reasons for this:
1. when initially trying to get something to work, I would prefer to just run the command interactively so I can see if there are any errors before jumping right in to having it supervised. if you recommend exactly what command to run from the command line including whatever user and environment needs to be put in place, then it is trivial to do this initial debug running, and I am then in a good position to setup supervision using whatever system I prefer.

This is one respect in which the runit/daemontools "run script" approach is really helpful. If you install that run script as recommended, you can it (and runit services in general) in the foreground at a command prompt.

cd /etc/sv/arvados-git-httpd
./run

1. Put script in /var/sv/name-of-service/run, chmod +x it, do whatever else you think you'll need
2. cd /var/sv/name-of-service && ./run in a terminal, see what happens, leave it open while you watch logs, etc.
3. When you're happy, ^C the one in your terminal and do ln -s /var/sv/name-of-service /etc/sv/
4. Now make sure it's alive, logging in the desired place, etc.
5. Want to un-supervise it so you can hack in a terminal? rm /etc/sv/name-of-service; sv exit /var/sv/name-of-service and you'll be right back to step 2: runit won't be fighting with you over the port number etc.

2. if we are already using another system (like supervisord) to manage processes and prefer to keep doing that (e.g. because we already have monitoring dashboards setup for it) and want to continue using that, we now have to learn enough of the runit syntax in order to figure out how to transcode them into supervisord (rather than you just telling us what is actually required without encoding it in runit files).

• use of chpst to set user ID (could change to sudo)
• use of chpst to set environment vars (ditto)
• we don't mention that the "run" script is just a shell program -- you can invoke it via runit, a terminal window, a different daemon supervisor, ....

We also don't mention the fact that logging goes to stdout so (if you're not using the given runit logging setup) you should attach stdout to something useful.

3. there seems to be some heterogeneity in runit versions and support of some features, and you could avoid those issues by not discussing it.

Did you run into this sort of trouble with the existing scripts? Even when we drop runit as a dependency, we do still want to offer runit scripts for the people who want them, and we'll probably start by packaging the scripts that are currently in the docs.

Tom Clegg wrote:

Brett Smith wrote:

• I'm skeptical about the suggested proxy headers. I know for sure X-External-Client isn't appropriate, because that's specific to the API server, and requires additional nginx configuration. We're currently not setting any headers in production, so I think we can do without them here as well.

Indeed, removed the extra headers and options that aren't relevant.

Sorry, I was not specific enough. I only meant to refer to the extra headers set with proxy_set_headers. The timeouts and other settings we do have in production. I don't know if we've done extensive testing on what "good" values are for the other settings for arv-git-httpd, but until we do I think it's probably safest to keep those and have everyone on the same page.

I think you're good to merge with that change. Josh's points are well-taken, and there's absolutely room for further improvement here, but the goal of this story is to document our git setup as it exists today. Figuring out the right way to deal with runit and other related infrastructure will take a little more planning and consideration. I will file tickets about those and link them here. In the meantime, this is a worthwhile improvement. Thanks.

Brett Smith wrote:

Figuring out the right way to deal with runit and other related infrastructure will take a little more planning and consideration.

This is #6884. I realized there are a lot of questions to settle here, so I made a planning story for us to answer those and craft actual implementation stories. Josh, you've mentioned using supervisord and having dashboards attached to those. If you could explain a little more about your setup and what would be most helpful to you in the packages, that'd be a big help.

(I also filed #6883 for Tom's earlier suggestion to have arvados-api-server-upgrade.sh set up git_repositories_dir.)

Brett Smith wrote:

Sorry, I was not specific enough. I only meant to refer to the extra headers set with proxy_set_headers. The timeouts and other settings we do have in production. I don't know if we've done extensive testing on what "good" values are for the other settings for arv-git-httpd, but until we do I think it's probably safest to keep those and have everyone on the same page.

OK, except I'm refusing to add back "proxy_redirect off". That merely ensures that an absolute redirect cannot work, because git-http-backend has no way of knowing what host:port will work for the client. (As far as I know git-http-backend never redirects at all, in which case it doesn't matter what we do here.)

Tom Clegg wrote:

Brett Smith wrote:

Sorry, I was not specific enough. I only meant to refer to the extra headers set with proxy_set_headers. The timeouts and other settings we do have in production. I don't know if we've done extensive testing on what "good" values are for the other settings for arv-git-httpd, but until we do I think it's probably safest to keep those and have everyone on the same page.

OK, except I'm refusing to add back "proxy_redirect off". That merely ensures that an absolute redirect cannot work, because git-http-backend has no way of knowing what host:port will work for the client. (As far as I know git-http-backend never redirects at all, in which case it doesn't matter what we do here.)

Fair. I think we're all set now.