Arvados

Slight complication: you can't necessarily fetch a commit from a remote by commit sha1. The server gets to decide whether to support that, or whether you need to know a tag or branch of which the commit is an ancestor.

We already ran into this on #12107#note-11 and sure enough pulling a commit hash from arv-git-httpd doesn't work, at least in the integration test environment.

• make arv-git-httpd support fetching by sha1 (perhaps it's easy with a git/git-http-backend/gitolite config)
• pull/tag the desired commit into the internal.git repo when creating a container (similar to crunch1) and have crunch-job pull the tagged commit from internal.git

Tom Clegg wrote:

• make arv-git-httpd support fetching by sha1 (perhaps it's easy with a git/git-http-backend/gitolite config)

This (git 2.5+) seems promising

[uploadpack]
        allowReachableSHA1InWant = true

But we should also handle the more general case where that's not enabled, so enabling that automatically on arvados-hosted repos seems desirable but not a blocker.

• start arv-git-httpd and nginx proxy earlier in run-tests (used to only bring these up for workbench tests)
• add git https address to discovery doc
• todo: advise Upgrading to master
• arv-git-httpd can access repos by UUID, not just name (otherwise crunch-run would need to look up the name, which seems like a waste)
• todo: clean up & test this in arv-git-httpd
• crunch-run can check out a git tree
• tests for the fetch-and-checkout code, using arv-git-httpd backend (added some fake data for "repository2" and removed some extra cruft from the tarball like sample git hooks)
• todo: add a stanza in TestSetupMounts, too
• todo: validate commit actually looks like a sha1
• defer/avoid?: currently apiserver does not try to stay in sync with crunch-run wrt validating mounts -- it just resolves UUID to PDH for collection mounts and ignores everything else. We could duplicate the crunch-run validations (path=/, commit hash only, etc.) in apiserver so they get rejected when committing a container request instead of waiting until a node comes up. But if we're eager to get this hooked up for a use case where our own code is writing the mounts, maybe we shouldn't block a merge behind this?
• todo: update docs

FAIL: git_mount_test.go:126: GitMountSuite.TestInvalid

git_mount_test.go:160:
    c.Check(err, check.ErrorMatches, trial.matcher)
... value = nil
... regex string = ".*sha1.*" 
... Error value is nil

git_mount_test.go:160:
    c.Check(err, check.ErrorMatches, trial.matcher)
... value = nil
... regex string = ".*UUID.*" 
... Error value is nil

• rebased on master
• docs updated
• cleanup + tests for the arv-git-httpd change
• test git_tree mounts in TestSetupMounts
• validate mount doesn't try anything unsupported
• fix an old bug and remove an old debug printf in arv-git-httpd
• todo?: Both apiserver and crunch-run need to be updated in order for this to work. Does that even deserve to be mentioned on Upgrading to master?
• todo: fix api tests (broken by changes to test git tarball and run-tests)

8311-mount-git @ d5f8662b50e80d8ffbeaff61d1c7dfb617e81298

https://ci.curoverse.com/job/developer-run-tests/538/

crunch-run error:

While setting up mounts: git fetch "http://192.168.5.2:9001/2tlax-s0uqq-u6kz3a8x06tczjb.git": unexpected client error: unexpected requesting "http://192.168.5.2:9001/2tlax-s0uqq-u6kz3a8x06tczjb.git/info/refs?service=git-upload-pack" status code: 500

arv-git-httpd error:

2017-12-14_19:49:34.67766 2017/12/14 19:49:34 "192.168.5.2:60622" "<invalid>" 500 "arvados API server error: Not logged in (401: 401 Unauthorized) returned by 192.168.5.2:8000" "2tlax-s0uqq-u6kz3a8x06tczjb" "GET" "/2tlax-s0uqq-u6kz3a8x06tczjb.git/info/refs"

Hmm, this looks like it could be a problem, eh?

    err = repo.Fetch(&git.FetchOptions{
        RemoteName: "origin",
        Auth:       git_http.NewBasicAuth("none", arvadostest.ActiveToken),
    })

2017-12-14T20:27:15.508185948Z 2017-12-14 20:27:15 cwltool INFO: /usr/bin/arvados-cwl-runner 0bb435a47e427b12fa2351141a22a1ba1e28a49d 1.0.20171010180436, arvados-python-client 0.1.20171010180436, cwltool 1.0.20170928192020
2017-12-14T20:27:15.512156671Z 2017-12-14 20:27:15 cwltool INFO: Resolved '/var/lib/cwl/run/revsort.cwl' to 'file:///var/lib/cwl/run/revsort.cwl'
2017-12-14T20:27:15.512520925Z 2017-12-14 20:27:15 cwltool ERROR: Tool definition failed initialization:
2017-12-14T20:27:15.512520925Z [Errno 13] Permission denied: '/var/lib/cwl/run/revsort.cwl'

I'm guessing this means the mountpoint isn't readable in the container because the mktemp dir isn't group/world-readable and container UID != crunch-run UID. Working on test & fix for that.

8311-mount-git @ e64b71e098878da66c434b7f7c09d36f2650f38e

This works! Composer 'run' button (12060-run-workflow) now submits container request using git repo mount, and it runs successfully in arvbox.

• fixes python tests

8311-mount-git @ e64b71e098878da66c434b7f7c09d36f2650f38e LGTM

Except all the tests need to pass, of course.