Support #16558

Document FS structure & supported features of WebDAV and S3 compatible APIs

Added by Peter Amstutz 4 months ago. Updated 3 days ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Documentation
Target version:
Start date:
10/15/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

For the audience of people trying to automate getting data into and out of keep using our WebDAV or S3 APIs.

Currently, there is a lot of info at https://godoc.org/git.arvados.org/arvados.git/services/keep-web but it needs to be more accessible from the SDK section of https://doc.arvados.org/.


Subtasks

Task #16967: Review 16558-keep-web-apiResolvedTom Clegg

Associated revisions

Revision e46e9a2a (diff)
Added by Peter Amstutz 6 days ago

Add WebDAV and S3 API documentation

Remove sections of keep-web/doc.go are now part of the API
documentation

refs #16558

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

Revision 948c9603 (diff)
Added by Peter Amstutz 6 days ago

Add WebDAV and S3 API documentation

Remove sections of keep-web/doc.go are now part of the API
documentation

refs #16558

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <>

History

#1 Updated by Peter Amstutz 4 months ago

  • Tracker changed from Bug to Support

#2 Updated by Peter Amstutz 4 months ago

  • Subject changed from Document Keep APIs (keepstore, webdav, upcoming s3) to Document Keep APIs (keepstore, webdav)

#3 Updated by Peter Amstutz 4 months ago

  • Description updated (diff)
  • Subject changed from Document Keep APIs (keepstore, webdav) to Document webdav

#4 Updated by Peter Amstutz 4 months ago

  • Target version changed from 2020-07-15 to 2020-08-12 Sprint

#5 Updated by Tom Clegg 3 months ago

  • Description updated (diff)

#6 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint

#7 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint

#8 Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint

#9 Updated by Peter Amstutz about 1 month ago

  • Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint

#10 Updated by Peter Amstutz 29 days ago

  • Target version changed from 2020-10-07 Sprint to 2020-10-21 Sprint

#11 Updated by Peter Amstutz 16 days ago

  • Subject changed from Document webdav to Document FS structure & supported features of WebDAV and S3 compatible APIs

#12 Updated by Peter Amstutz 16 days ago

  • Description updated (diff)

#13 Updated by Peter Amstutz 16 days ago

  • Category set to Documentation

#14 Updated by Peter Amstutz 15 days ago

  • Assigned To set to Peter Amstutz

#15 Updated by Peter Amstutz 13 days ago

  • Status changed from New to In Progress

#16 Updated by Peter Amstutz 8 days ago

16558-keep-web-api @ 6c8ca0862ed8a66dd42a85ec498fd6e32fad310f

First pass at documenting WebDAV and S3. Focus is on documenting supported features and authorization methods.

#17 Updated by Tom Clegg 7 days ago

Yikes, moving the keep-web commentary out of godoc is long overdue. Thanks.

s3

For GetBucketVersioning, "this is a stub to avoid breaking clients" seems like a weird way to put it, and sort of implies it's incomplete, which it isn't. Maybe "Bucket versioning is not supported, so the response from this API always indicates bucket versioning has never been enabled."? (I don't think we need to justify "why have an API that always returns the same thing? to avoid breaking clients" -- in this context it seems implicit/obvious enough.)

Accepts AWS Signature Version 4 format Authorization: AWS AWS4-HMAC-SHA256 Credential=...,SignedHeaders=...,Signature=...

Delete "AWS" there (should be Authorization: AWS4-HMAC-SHA256 ...) ... but see below

The "Credential" is the Arvados token uuid,

Credential is {AccessKey}/{Scope}, not just AccessKey ... but ...

Maybe we should back off a bit from detailing the V4 Authorization header format here. Better to just explain how to map token UUID/secret to AccessKey/SecretKey, and defer to AWS docs / client libraries for the signing methods?

the signature can be anything as keep-web will ignore it.

Similarly, describing the header instead of the AccessKey/SecretKey mapping is kind of awkward because client libraries ask for a SecretKey, not a signature -- "the SecretKey can be anything" is probably the information they need.

Perhaps something along these lines would be easiest to follow?
  • If your client uses V4 signatures exclusively, use the Arvados token's UUID part as AccessKey, and its secret part as SecretKey. This is more common, and recommended.
  • If your client uses V2 signatures -- or a combination of V2 and V4 -- use the Arvados token's secret part as both AccessKey and SecretKey.

(As an aside, I think we could improve the V2 code so we accept UUID/secret for both V2 and V4 ... it's just that it doesn't seem too urgent because clients generally support V4. It sure would simplify this section for people who don't know/care which kind of signature their client is using, though.)

webdav

The first part of the webdav page looks pretty good.

The "It can be installed anywhere" part seems out of place. I think we already deal with install concerns in the install docs.

"Fetching the root path ... return a 401" sounds a bit weird. Maybe better to start off with "Requests can be authenticated using Basic or Bearer", then (if needed?) mention the RFC 7617 401 thing. This point should probably be in an "authentication" section instead of "browsing". (Also, I'm not 100% sure but I think we use the anonymous token instead of returning 401 at the root URL, if anonymous token is configured.)

Perhaps worth mentioning that webdav also works with /c=foo/t=bar/ and {uuid-or-pdh}.configured.domain.example?

keep-web

Starting at the "URL structure" heading, the webdav page isn't really about webdav any more. Maybe the "other stuff you can do with keep-web" part should be a separate page?

"Authorization mechanisms" heading should really be "Authentication mechanisms", right? (my bad)

There's a spurious "<" at the end of the query string api_token example.

Maybe we should use v2 tokens in these examples so they look more familiar.

We should probably delete the godoc copy of the "stuff keep-web can do" essay, and replace with a link to the new doc page. Keeping divergent copies seems iffy.

#18 Updated by Peter Amstutz 6 days ago

16558-keep-web-api @ 817d7985e27cf188195fed1c7c3e412fb53beb5b

Tweak S3 docs and move keep-web URL patterns to its own page

(Also, I'm not 100% sure but I think we use the anonymous token instead of returning 401 at the root URL, if anonymous token is configured.)

I'm not sure if this is true, in my experience if you arrived at a keep-web link unauthorized, it generally pops up the basic authentication dialog box.

#19 Updated by Tom Clegg 6 days ago

Looks like keep-web-urls.html.textile.liquid didn't get git-added, but the rest LGTM, thanks!

#20 Updated by Peter Amstutz 3 days ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF