Arvados

Nico's Whishlist of sysadmin stuff to have documented:

• diagram of suggested VM <-> services.
• SSL certificates: which ones should deploy where? how many do we need
• User management: how do I give permissions to users? what is a good group strategy ? (just explaining the Permission Models is 10% of the job)
• Docker images, the how to use any docker image and most important is why we have to store it in keep. Permissions related to it.
• Storing data: Defining keepproxy, and what is "in" and "out" of the cluster
• Storing data: arv-put lifecycle and where the data is store, explaining that manifests and data chunks are in different places and why. de-duplication how it works. and why do I have to load twice when the data is already there?
• Running pipelines: from CWL files -> dispatcher -> slurm -> output of the workflow. where is everything and where should I look into for debugging when things don't work (queued for ever, failed pipelines, limits reached, etc)
• managing nodes in the cloud: concept of "arvados compute image" and what is needed in a compute node, configurations bits of a-n-m and tuning on them.
• Scalability/fine tuning parameters: connections to database, concurrency in the different components, size of the VMs for services etc.
• resources in arvados: arv edit and it's friends. all the possible things we do as admins: create keepstore servers, shell nodes, user permissions, permission links, docker image permission group, git repositories etc
• Tokens. Everything about them, what is a scope? what do I need for common use cases (dispatcher, shell users, workbench, etc)? what is an api_client, api_client authorization and is it related to a token at all? what is a supertoken and when is good to use it? what is anonymous tokens and where to use it?
• Keep web an link exports: how can I just use a URL to export some data from Keep? what are the security risks ? can keep-web use a super token ?

Also fix https://doc.arvados.org/sdk/go/index.html

13282-admin-docs @ c32c841b5718ea1c5e594e015b2376addc5af40a

• Add "architecture" and "admin" sections
• Add a page listing all the Arvados components with a brief description of each
• Add a section about platform support under "manual install prerequisites"
• Comment out pages which describe how to do things with crunch v1 to reduce confusion (most of these pages already had deprecation notices on them)
• Fix a few code blocks with rendering glitches caused by {% comment %} sections

• A number of warnings during the build, which would be useful to get rid of so that important future ones aren't ignored.
  The most serious of which seems to be more than a warning: Warning: R not found, R documentation will not be generated

• Don't remove documentation for deprecated APIs until the APIs themselves are removed. Moving them to an "Obsolete" or "Deprecated" section is fine, but for Crunch 1 in particular, we haven't even described to users how to migrate to Crunch 2 yet.

• Space character in filename for "Arvados arch.odg" seems like asking for trouble

Tom Morris wrote:

• A number of warnings during the build, which would be useful to get rid of so that important future ones aren't ignored.

I cleaned up some of the R documentation ones. For Python, there's probably a whole story worth of cleaning up the documentation (such as divesting from epydoc and using something like sphinx).

The most serious of which seems to be more than a warning: Warning: R not found, R documentation will not be generated

If it can't find R, it can't generate the R documentation. Do you think this should be a fatal error instead of a warning?

• Don't remove documentation for deprecated APIs until the APIs themselves are removed. Moving them to an "Obsolete" or "Deprecated" section is fine, but for Crunch 1 in particular, we haven't even described to users how to migrate to Crunch 2 yet.

Ok, those pages are now part of an "Obsolete" section of the user guide.

• Space character in filename for "Arvados arch.odg" seems like asking for trouble

Renamed (but that file is only there in case anyone wants to edit the diagram, the actual image used in the documentation is exported to Arvados_arch.svg)