Arvados

Ready for review at 8a27fe370239ecb8e50d53f46b45ed61203a35ca on branch 19175-doc-refactor-multi-host-installation,

arvados/install/salt.html¶

"we have developed a Saltstack ’s arvados-formula which can "

Stray ’s

h2(#provisioning_tool_files and directories). The provisioning tool files and directories

Formatting error

cp local.params.example.multiple_hosts local.params
cp -r config_examples/multi_host/aws local_config_dir

These commands are repeated in both the "Provisioning Arvados with Saltstack" and the "Choose an Arvados installation configuration" sections, and then repeated again on the individual single/multi host install pages.

Dump the configuration files created with the provision script

I believe this is optional and just informative, but the way it is written is imperative (something the user needs to do). It should say "How to dump..." or "Dumping..."

Also should clarify this is the installer's configuration not the Arvados config file (although I guess the arvados config is derived from it).

h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure

another formatting error

arvados/install/salt-vagrant.html¶

I don't think we're supporting the vagrant install, in any event it should probably go after the single/multi node install in the docs with a note that it's really only intended for development.

/install/salt-single-host.html¶

it uses the local machine disk for Keep storage (under the /tmp directory).

Does it really put keep blocks in /tmp ?????

which can run just eight concurrent CWL jobs

Rephrase "has a limit of eight concurrent jobs".

git checkout main

Shouldn't this be "git checkout 2.4-release"

Actually, this should '2.4-release' in release versions and 'main' on main. There's a few places in the docs where we want that, instead of updating them all manually it should be a variable substitution.

Liquid error: No such template ‘ssl_config’

Uh oh?

scp -r provision.sh local* tests user@host:

So, we don't state explicitly that you're should be doing the git clone and config file editing on your local workstation. People who don't use Linux locally are more likely to start on the VM itself. In which case the scp step is unnecessary and potentially confusing.

The key thing that needs to be stated clearly is that provision.sh is supposed to run on the host where everything will be installed.

arvados/install/salt-multi-host.html¶

Copy the configuration files

Maybe say "Copy and edit" or "Copy and customize" since they have to be updated.

If you try to run provision.sh without editing the file, will it give an error saying you didn't update stuff, or just crash?

Could the provision script have a subcommand for "copy self to target host, then run" ?

Could the provision script have a subcommand to distribute a new arvados config.yml to all the hosts?

Peter Amstutz wrote:

arvados/install/salt.html¶

"we have developed a Saltstack ’s arvados-formula which can "

Stray ’s

h2(#provisioning_tool_files and directories). The provisioning tool files and directories

Formatting error

cp local.params.example.multiple_hosts local.params
cp -r config_examples/multi_host/aws local_config_dir

These commands are repeated in both the "Provisioning Arvados with Saltstack" and the "Choose an Arvados installation configuration" sections, and then repeated again on the individual single/multi host install pages.

Dump the configuration files created with the provision script

I believe this is optional and just informative, but the way it is written is imperative (something the user needs to do). It should say "How to dump..." or "Dumping..."

Also should clarify this is the installer's configuration not the Arvados config file (although I guess the arvados config is derived from it).

h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure

another formatting error

Ah, I had not touched this page as part of this branch. I think it is superfluous so I just removed it. The only bit on there that was vaguely interesting (for developing/debugging) was the dump mode for the provision script, but that script already prints out something similar in its help text for the --dump-config flag.

arvados/install/salt-vagrant.html¶

I don't think we're supporting the vagrant install, in any event it should probably go after the single/multi node install in the docs with a note that it's really only intended for development.

Same, I had not touched this page as part of this branch. We don't use this and don't support it so I just unlinked the page from the menu for now. We could also just remove the file from the docs if you prefer.

/install/salt-single-host.html¶

it uses the local machine disk for Keep storage (under the /tmp directory).

Does it really put keep blocks in /tmp ?????

Yes (cf. config_examples/single_host/single_hostname/pillars/arvados.sls), it does not make assumptions about being somewhere with block storage. The very first paragraph on this page about the limitations of this setup suggest changing the config for a different storage volume.

which can run just eight concurrent CWL jobs

Rephrase "has a limit of eight concurrent jobs".

Done.

git checkout main

Shouldn't this be "git checkout 2.4-release"

Actually, this should '2.4-release' in release versions and 'main' on main. There's a few places in the docs where we want that, instead of updating them all manually it should be a variable substitution.

That is already the case. The branchname variable is filled in automatically (see _includes/_branchname.liquid) based on the site.current_version parameter which is set by the build scripts.

Liquid error: No such template ‘ssl_config’

Uh oh?

Ah, sorry fixed.

scp -r provision.sh local* tests user@host:

So, we don't state explicitly that you're should be doing the git clone and config file editing on your local workstation. People who don't use Linux locally are more likely to start on the VM itself. In which case the scp step is unnecessary and potentially confusing.

Yes, though, even on Windows you can do git clone now via WSL2. The text above that line did say "copy the files (if needed)". Anyway; I've added some more wording to address this. It's best practice to do the checkout on your workstation, and then copy things over before executing the provision script there. That way it is safe to blow away the target host and start over as needed.

The key thing that needs to be stated clearly is that provision.sh is supposed to run on the host where everything will be installed.

Yes, I've modified the text.

arvados/install/salt-multi-host.html¶

Copy the configuration files

Maybe say "Copy and edit" or "Copy and customize" since they have to be updated.

Done.

If you try to run provision.sh without editing the file, will it give an error saying you didn't update stuff, or just crash?

It tells you you need to modify parameters:

The config file /personal/projects/arvados/arvados/tools/salt-install/local.params has some parameters that need to be modified.
Please, fix them and re-run the provision script.
Error occurred (1) while running ./provision.sh at line 1 : exit 1

Could the provision script have a subcommand for "copy self to target host, then run" ?

I suppose it could.

Could the provision script have a subcommand to distribute a new arvados config.yml to all the hosts?

That's harder because it does not know about all your hosts. Every invocation of the script only knows about installing whatever you ask it to on whatever node you ask it to. Also; I think we do not want this; remember we advise people to stop running the installer once their cluster is up, cf. the last line on the single and multi page:

Re-running the Salt-based installer is not recommended for maintaining and upgrading Arvados, please see Maintenance and upgrading for more information.

Anyway; I do have another script to distribute config.yml, once the set of hosts is known. I could put that somewhere (where?).

Ready for another look at 8e2d796199b51ea5148c77be1a763fee96947a88 on branch 19175-doc-refactor-multi-host-installation

LGTM