Arvados

9957-keep-web-config @ caba74a

Keep-web provides read-only HTTP access to files stored in Keep; see
https://godoc.org/github.com/curoverse/arvados/services/keep-web and
http://doc.arvados.org/install/install-keep-web.html

Usage: keep-web -config path/to/config.json

Options:
  -allow-anonymous
        Load an anonymous token from the ARVADOS_API_TOKEN environment variable (DEPRECATED -- use config file instead)
  -attachment-only-host host:port
        Only serve attachments at the given host:port (DEPRECATED -- use config file instead)
  -config path
        path to json configuration file (default "/etc/arvados/keep-web/config.json")
  -listen string
        address:port or :port to listen on (DEPRECATED -- use config file instead)
  -trust-all-content
        Serve non-public content from a single origin. Dangerous: read docs before using! (DEPRECATED -- use config file instead)

Example config file:
    {
      "Client": {
        "APIHost": "zzzzz.arvadosapi.com:443",
        "AuthToken": "",
        "Insecure": false
      },
      "Listen": ":80",
      "AnonymousTokens": [
        "xxxxxxxxxxxxxxxxxxxxxxx" 
      ],
      "AttachmentOnlyHost": "",
      "TrustAllContent": false
    }

Client.APIHost:

    Address (or address:port) of the Arvados API endpoint.

Client.AuthToken:

    Should be empty.

Client.Insecure:

    True if your Arvados API endpoint uses an unverifiable SSL/TLS
    certificate.

Listen:

    Local port to listen on. Can be "address", "address:port", or
    ":port", where "address" is a host IP address or name and "port" 
    is a port number or name.

AnonymousTokens:

    Array of tokens to try when a client does not provide a token.

AttachmentOnlyHost:

    Accept credentials, and add "Content-Disposition: attachment" 
    response headers, for requests at this hostname:port.

    This prohibits inline display, which makes it possible to serve
    untrusted and non-public content from a single origin, i.e.,
    without wildcard DNS or SSL.

TrustAllContent:

    Serve non-public content from a single origin. Dangerous: read
    docs before using!

Comments at config:1d5e807

• Can we call sdk/configfile as sdk/config?

• The usage says, config.AuthToken should be empty. What happens if a user provides it in the file? Do we raise an error or ignore it?

• This “// The "AnonymousTokens" configuration entry is an array of tokens to use when clients try to retrieve files without providing their own Arvados API token” : I am wondering if this might confuse when read with the above about “config.AuthToken should be empty” as to how to send the token. Can you clarify that the api token in non-anonymous case would be part of the URL?

• 18 tests failing with run-tests (I did a reinstall)

----------------------------------------------------------------------
FAIL: handler_test.go:72: IntegrationSuite.TestVhostViaPath

doRequests: zzzzz-4zz18-fy296fx3hot09f7.example.com/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/_/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3+45.example.com/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3-45--collections.example.com/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-foonbarfilesdir.example.com/dir1/foo
handler_test.go:73:
    s.doVhostRequests(c, authzViaPath)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

----------------------------------------------------------------------
FAIL: handler_test.go:80: IntegrationSuite.TestVhostViaQueryString

doRequests: zzzzz-4zz18-fy296fx3hot09f7.example.com/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/_/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3+45.example.com/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3-45--collections.example.com/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-foonbarfilesdir.example.com/dir1/foo
handler_test.go:81:
    s.doVhostRequests(c, authzViaQueryString)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

----------------------------------------------------------------------
FAIL: handler_test.go:99: IntegrationSuite.TestVhostViaXHRPOST

doRequests: zzzzz-4zz18-fy296fx3hot09f7.example.com/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-fy296fx3hot09f7--collections.example.com/_/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3+45.example.com/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: 1f4b0bc7583c2a7f9102c395f4ffc5e3-45--collections.example.com/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

doRequests: zzzzz-4zz18-foonbarfilesdir.example.com/dir1/foo
handler_test.go:100:
    s.doVhostRequests(c, authzViaPOST)
handler_test.go:157:
    c.Check(body, check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

[keep1] 2016/09/19 12:18:23 [[::1]:47397] GET acbd18db4cc2f85cedef654fccc4a4d8+3+Ae08ce2ee119087057af1d1b6d036f5da641b0ac1@57f284cf 0 0.000163s 0.000150s 0.000013s 403 10 "Forbidden" 
[keep0] 2016/09/19 12:18:23 [[::1]:47601] GET acbd18db4cc2f85cedef654fccc4a4d8+3+Ae08ce2ee119087057af1d1b6d036f5da641b0ac1@57f284cf 0 0.000077s 0.000069s 0.000008s 403 10 "Forbidden" 
2016/09/19 12:18:23 "" "WARNING" "Our status changed from 200 to 502 after we sent headers" 
2016/09/19 12:18:23 "" 502 "GET acbd18db4cc2f85cedef654fccc4a4d8+3+Ae08ce2ee119087057af1d1b6d036f5da641b0ac1@57f284cf failed: [http://localhost:33633/acbd18db4cc2f85cedef654fccc4a4d8+3+Ae08ce2ee119087057af1d1b6d036f5da641b0ac1@57f284cf: HTTP 403 \"Forbidden\" http://localhost:45877/acbd18db4cc2f85cedef654fccc4a4d8+3+Ae08ce2ee119087057af1d1b6d036f5da641b0ac1@57f284cf: HTTP 403 \"Forbidden\"]" 0 "POST" "example.com" "/c=zzzzz-4zz18-fy296fx3hot09f7/foo" "" 

----------------------------------------------------------------------
FAIL: handler_test.go:401: IntegrationSuite.TestXHRNoRedirect

handler_test.go:420:
    c.Check(resp.Body.String(), check.Equals, "foo")
... obtained string = "" 
... expected string = "foo" 

Sent SIGTERM to 27608 (/home/radhika/arvados/tmp/keep0.pid)
[keep0] 2016/09/19 12:18:23 caught signal: terminated
[keep0] 2016/09/19 12:18:23 keepstore exiting, pid 27608
Sent SIGTERM to 27619 (/home/radhika/arvados/tmp/keep1.pid)
[keep1] 2016/09/19 12:18:23 caught signal: terminated
[keep1] 2016/09/19 12:18:23 keepstore exiting, pid 27619
OOPS: 8 passed, 18 FAILED

Radhika Chippada wrote:

• Can we call sdk/configfile as sdk/config?

Sure. I thought of it as being just about loading from the file (as opposed to other config stuff like defaults) but leaving room for stuff like config-by-network and update-config-on-the-fly seems reasonable. Updated.

• The usage says, config.AuthToken should be empty. What happens if a user provides it in the file? Do we raise an error or ignore it?

Updated comment to say "Unused. Normally empty, or omitted entirely."

• This “// The "AnonymousTokens" configuration entry is an array of tokens to use when clients try to retrieve files without providing their own Arvados API token” : I am wondering if this might confuse when read with the above about “config.AuthToken should be empty” as to how to send the token. Can you clarify that the api token in non-anonymous case would be part of the URL?

Updated:

// The "AnonymousTokens" configuration entry is an array of tokens to
// use when processing anonymous requests, i.e., whenever a web client
// does not supply its own Arvados API token via path, query string,
// cookie, or request header.

Those updates are great. Thanks.

As we discussed, the tests only seem to be failing for me, even in master, so please ignore my comment about test failures. Thanks.