Arvados

Suggest the following behavior:

Prefer HTTPS if available by looking at clone_urls. Rationale: HTTPS is more likely to work than ssh because it only requires the API token and we already know it works. It is harder to reliable determine if SSH is going to work, need to probing the local ssh keys and the authorized_keys table).

Partial fix on branch 6605-arv-copy-http (uses http in preference to ssh, but doesn't fall back to trying ssh if http fails).

• Prefer HTTPS if available, try ssh if not available or HTTPS fails.
• Tell git not to ask for password

6605: Check to see if each git URL works by using git ls-remote.

Preferentially use https.

Set GIT_ASKPASS=/bin/false to suppress asking for credentials.

6605-arv-copy-http 5fb73e1

I'm not keen on editing the user's global git config here. This edit seems pretty safe, but it's still rather surprising as a silent side effect of running arv-copy. Could we easily detect whether the credential helper is already in place, and if not, set it up and say something on stderr? Or perhaps just use "-c credential.blah.helper=blah" whenever invoking git, and avoid editing config at all.

If we stick with editing configs, since we're going to try all the http_urls if >1 are given, how about

-    if http_url:
-        setup_git_http(http_url[0])
+    for url in http_url:
+        setup_git_http(url)

...or even do it in the existing "for url in" loop, so we only do it for the urls that we actually try using.

If you feel like adding scope (but not if you don't) we could detect whether the repository is https://github.com/foo/bar and skip all the copying stuff (it looks like currently it'll just crash "not found" in that case).

Tom Clegg wrote:

6605-arv-copy-http 5fb73e1

I'm not keen on editing the user's global git config here. This edit seems pretty safe, but it's still rather surprising as a silent side effect of running arv-copy. Could we easily detect whether the credential helper is already in place, and if not, set it up and say something on stderr? Or perhaps just use "-c credential.blah.helper=blah" whenever invoking git, and avoid editing config at all.

Thanks, "-c" is a much better solution, I didn't realize it was an option. Done.

If you feel like adding scope (but not if you don't) we could detect whether the repository is https://github.com/foo/bar and skip all the copying stuff (it looks like currently it'll just crash "not found" in that case).

Unfortunately it's not that easy, for this to work it also needs to do the right thing when setting the repository and script_version fields on the destination side, and that's when I decided the hole was looking a little bit dark and ratty.

Peter Amstutz wrote:

If you feel like adding scope (but not if you don't) we could detect whether the repository is https://github.com/foo/bar and skip all the copying stuff (it looks like currently it'll just crash "not found" in that case).

Unfortunately it's not that easy, for this to work it also needs to do the right thing when setting the repository and script_version fields on the destination side, and that's when I decided the hole was looking a little bit dark and ratty.

Ah. I thought "the right thing" would be nothing / leave them alone... but anyway, fine with punting.

I don't like conflating the existing meaning of ARVADOS_API_HOST_INSECURE ("accept any cert at https://api") with this new usage ("allow fetching/pushing repos using http://"). Perhaps an --allow-git-http flag would be more clear, and possible for a user to discover in some way other than trial & error?

Tom Clegg wrote:

I don't like conflating the existing meaning of ARVADOS_API_HOST_INSECURE ("accept any cert at https://api") with this new usage ("allow fetching/pushing repos using http://"). Perhaps an --allow-git-http flag would be more clear, and possible for a user to discover in some way other than trial & error?

Added --allow-git-http-src and --allow-git-http-dst instead of looking at ARVADOS_API_HOST_INSECURE. Now at e71b6d2