Arvados

So, our keepstore install documentation doesn't document using keepstore config files (it still describes the old way which is using command line flags). However StorageClasses can only be specified in the config file. So in order to document how the admin should configure StorageClasses without terribly confusing the reader is to first update the keepstore install documentation for config files.

8de0cca20 is an interesting experiment in getting the docs closer to the code, but I'm not sure whether it's really better in the end.

• Is there some way we can use traditional godoc- and editor-friendly struct field comments to generate docs?
• What's the most important place to offer the detailed information -- "keepstore -help", or doc.arvados.org, or...?

If we go with struct tags, instead of sort-of-parsing the struct tags with strings.Index(string(f.Tag), "doc:") etc., we can probably use f.Tag.Get("doc") ... https://golang.org/pkg/reflect/#StructTag.Get ... but (I'm guessing) only if we use proper struct tag syntax, `doc:"..."`, which presumably we should do anyway.

Tom Clegg wrote:

8de0cca20 is an interesting experiment in getting the docs closer to the code, but I'm not sure whether it's really better in the end.

Thoughts

• Is there some way we can use traditional godoc- and editor-friendly struct field comments to generate docs?

So the problem is, unlike struct Tags, comments aren't available at runtime, so we would need a separate tool to parse the file and generate the documentation, which seems a lot more complicated. I suppose it's possible there's already a tool out there that does something like this, I can do some googling.

• What's the most important place to offer the detailed information -- "keepstore -help", or doc.arvados.org, or...?

So the goal here actually is to offer detailed documentation on doc.arvados.org, but in a way that minimizes the friction of keeping it up to date (ideally it would even be part of the documentation generation process).

If we go with struct tags, instead of sort-of-parsing the struct tags with strings.Index(string(f.Tag), "doc:") etc., we can probably use f.Tag.Get("doc") ... https://golang.org/pkg/reflect/#StructTag.Get ... but (I'm guessing) only if we use proper struct tag syntax, `doc:"..."`, which presumably we should do anyway.

I tried that, it didn't like newlines in the tag value, which is why I did it that way.

a handy copy and paste oneliner is:

awk '/MemTotal/ { printf "%d\n", ($2 / (88 * 1024))}' /proc/meminfo

this could be added in the MaxBuffers description

Peter Amstutz wrote:

So the problem is, unlike struct Tags, comments aren't available at runtime, so we would need a separate tool to parse the file and generate the documentation, which seems a lot more complicated. I suppose it's possible there's already a tool out there that does something like this, I can do some googling.

Maybe something like https://github.com/robertkrimen/godocdown ?

So the goal here actually is to offer detailed documentation on doc.arvados.org, but in a way that minimizes the friction of keeping it up to date (ideally it would even be part of the documentation generation process).

If the goal is doc.arvados.org then the runtime introspection stuff seems like an unnecessary layer of challenges. Comments→docs is the way godoc etc. work, and seems more direct.

f.Tag.Get()

I tried that, it didn't like newlines in the tag value, which is why I did it that way.

Conflicting with standard tooling is a red flag. Sounds to me like struct tags are not a good fit here.

13579-keepstore-config-doc @ 296ac256c898c145f7e4566d86d5f09f4e966b43

Rebased and removed changes to keepstore.

Tom Morris wrote:

doc/install/install-keepstore.html.textile.liquid¶

• need a discussion of how to decide on number and size of Keepstore servers - redundancy, network & disk I/O bandwidth, disk capacity, RAM, etc

Agree, but Ward or Nico would be better qualified to write this.

Configure storage volumes¶

• supported volume types would scan better as a list
• local inline vs Azure & S3 on their own pages feels asymetric

Sure, should move the "local" config to its own page then?

modern disk controllers know how to reorder requests to minimize seek times, but they can only do that when they have visibility into the entire set of pending requests.

That was partly copied from some earlier text. I'll just make it more vague, like, may be useful "if you are experiencing contention with simultaneous requests to the same volume".

doc/install/configure-s3-object-storage¶

• As an alternative to local and network-attached POSIX filesystems,- Keepstore can store data in object storage compatible with the S3 AP
• What are possible values for Endpoint ?

It's relevant for GCP / RADOS but not for S3, will try to tweak the documentation for that.

• 10 minutes seems like a very long default timeout for operations

I think that's what keepstore's default is but I agree that is excessive.

This seems out of scope, but just in case you want to deal with it anyway... The keepstore package enables keepstore as a systemd service, which the docs should acknowledge (like https://doc.arvados.org/install/install-ws.html?). On debian and ubuntu I suspect this version (still) leaves you with a systemd service and a runit service competing for port 25107.

Rather than maintaining its own differently-formatted version of the install docs (which already seems to be out of sync) perhaps the keepstore -help usage message should offer a link to the docs.

Nit: the word "actual" seems superfluous in "The actual directory that will be used".

13579-keepstore-config-doc now @ 679d5507d6c390fd03e5c71300001f2696b1f5b1

Incorporated comments from above. Added documentation page about storage classes to "Admin" section (which is what this ticket was originally about).

added d9c1f53cd 13579-keepstore-config-doc for Peter to improve.

13579-keepstore-config-doc @ 18e820bdfe2598ec1901fd52fd80c8dae4f55c32

Covers #13580 because they are closely related. Added section about setting storage classes to user guide. Admin page links to it.

LGTM