https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422022-06-02T15:36:16ZArvadosArvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041062022-06-02T15:36:16ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041172022-06-02T17:40:40ZWard Vandewegeward@curii.com
<ul></ul><p>Ready for review at <a class="changeset" title="Refactor the multi-host salt install page. refs #19175 Arvados-DCO-1.1-Signed-off-by: Ward Vand..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/8a27fe370239ecb8e50d53f46b45ed61203a35ca">8a27fe370239ecb8e50d53f46b45ed61203a35ca</a> on branch 19175-doc-refactor-multi-host-installation,</p> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041212022-06-02T19:10:24ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><a name="arvadosinstallsalthtml"></a>
<h2 >arvados/install/salt.html<a href="#arvadosinstallsalthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>"we have developed a Saltstack ’s arvados-formula which can "</p>
</blockquote>
<p>Stray ’s</p>
<blockquote>
<p><code>h2(#provisioning_tool_files and directories). The provisioning tool files and directories</code></p>
</blockquote>
<p>Formatting error</p>
<blockquote>
<p>cp local.params.example.multiple_hosts local.params<br />cp -r config_examples/multi_host/aws local_config_dir</p>
</blockquote>
<p>These commands are repeated in both the "Provisioning Arvados with Saltstack" and the "Choose an Arvados installation configuration" sections, and then repeated <em>again</em> on the individual single/multi host install pages.</p>
<blockquote>
<p>Dump the configuration files created with the provision script</p>
</blockquote>
<p>I believe this is optional and just informative, but the way it is written is imperative (something the user needs to do). It should say "How to dump..." or "Dumping..."</p>
<p>Also should clarify this is the <em>installer's</em> configuration not the Arvados config file (although I guess the arvados config is derived from it).</p>
<blockquote>
<p><code>h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure</code></p>
</blockquote>
<p>another formatting error</p>
<a name="arvadosinstallsalt-vagranthtml"></a>
<h2 >arvados/install/salt-vagrant.html<a href="#arvadosinstallsalt-vagranthtml" class="wiki-anchor">¶</a></h2>
<p>I don't think we're supporting the vagrant install, in any event it should probably go after the single/multi node install in the docs with a note that it's really only intended for development.</p>
<a name="installsalt-single-hosthtml"></a>
<h2 >/install/salt-single-host.html<a href="#installsalt-single-hosthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>it uses the local machine disk for Keep storage (under the /tmp directory).</p>
</blockquote>
<p>Does it really put keep blocks in /tmp ?????</p>
<blockquote>
<p>which can run just eight concurrent CWL jobs</p>
</blockquote>
<p>Rephrase "has a limit of eight concurrent jobs".</p>
<blockquote>
<p>git checkout main</p>
</blockquote>
<p>Shouldn't this be "git checkout 2.4-release"</p>
<p>Actually, this should '2.4-release' in release versions and 'main' on main. There's a few places in the docs where we want that, instead of updating them all manually it should be a variable substitution.</p>
<blockquote>
<p>Liquid error: No such template ‘ssl_config’</p>
</blockquote>
<p>Uh oh?</p>
<blockquote>
<p>scp -r provision.sh local* tests user@host:</p>
</blockquote>
<p>So, we don't state explicitly that you're should be doing the git clone and config file editing on your local workstation. People who don't use Linux locally are more likely to start on the VM itself. In which case the <code>scp</code> step is unnecessary and potentially confusing.</p>
<p>The key thing that needs to be stated clearly is that provision.sh is supposed to run on the host where everything will be installed.</p>
<a name="arvadosinstallsalt-multi-hosthtml"></a>
<h2 >arvados/install/salt-multi-host.html<a href="#arvadosinstallsalt-multi-hosthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>Copy the configuration files</p>
</blockquote>
<p>Maybe say "Copy and edit" or "Copy and customize" since they have to be updated.</p>
<p>If you try to run provision.sh without editing the file, will it give an error saying you didn't update stuff, or just crash?</p>
<p>Could the provision script have a subcommand for "copy self to target host, then run" ?</p>
<p>Could the provision script have a subcommand to distribute a new arvados config.yml to all the hosts?</p> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041242022-06-02T20:41:27ZWard Vandewegeward@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<a name="arvadosinstallsalthtml"></a>
<h2 >arvados/install/salt.html<a href="#arvadosinstallsalthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>"we have developed a Saltstack ’s arvados-formula which can "</p>
</blockquote>
<p>Stray ’s</p>
<blockquote>
<p><code>h2(#provisioning_tool_files and directories). The provisioning tool files and directories</code></p>
</blockquote>
<p>Formatting error</p>
<blockquote>
<p>cp local.params.example.multiple_hosts local.params<br />cp -r config_examples/multi_host/aws local_config_dir</p>
</blockquote>
<p>These commands are repeated in both the "Provisioning Arvados with Saltstack" and the "Choose an Arvados installation configuration" sections, and then repeated <em>again</em> on the individual single/multi host install pages.</p>
<blockquote>
<p>Dump the configuration files created with the provision script</p>
</blockquote>
<p>I believe this is optional and just informative, but the way it is written is imperative (something the user needs to do). It should say "How to dump..." or "Dumping..."</p>
<p>Also should clarify this is the <em>installer's</em> configuration not the Arvados config file (although I guess the arvados config is derived from it).</p>
<blockquote>
<p><code>h2.(#add_formula_to_saltstack). Add the Arvados formula to your Saltstack infrastructure</code></p>
</blockquote>
<p>another formatting error</p>
</blockquote>
<p>Ah, I had not touched this page as part of this branch. I think it is superfluous so I just removed it. The only bit on there that was vaguely interesting (for developing/debugging) was the dump mode for the provision script, but that script already prints out something similar in its help text for the <code>--dump-config</code> flag.</p>
<blockquote>
<a name="arvadosinstallsalt-vagranthtml"></a>
<h2 >arvados/install/salt-vagrant.html<a href="#arvadosinstallsalt-vagranthtml" class="wiki-anchor">¶</a></h2>
<p>I don't think we're supporting the vagrant install, in any event it should probably go after the single/multi node install in the docs with a note that it's really only intended for development.</p>
</blockquote>
<p>Same, I had not touched this page as part of this branch. We don't use this and don't support it so I just unlinked the page from the menu for now. We could also just remove the file from the docs if you prefer.</p>
<blockquote>
<a name="installsalt-single-hosthtml"></a>
<h2 >/install/salt-single-host.html<a href="#installsalt-single-hosthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>it uses the local machine disk for Keep storage (under the /tmp directory).</p>
</blockquote>
<p>Does it really put keep blocks in /tmp ?????</p>
</blockquote>
<p>Yes (cf. <code>config_examples/single_host/single_hostname/pillars/arvados.sls</code>), it does not make assumptions about being somewhere with block storage. The very first paragraph on this page about the limitations of this setup suggest changing the config for a different storage volume.</p>
<blockquote><blockquote>
<p>which can run just eight concurrent CWL jobs</p>
</blockquote>
<p>Rephrase "has a limit of eight concurrent jobs".</p>
</blockquote>
<p>Done.</p>
<blockquote><blockquote>
<p>git checkout main</p>
</blockquote>
<p>Shouldn't this be "git checkout 2.4-release"</p>
<p>Actually, this should '2.4-release' in release versions and 'main' on main. There's a few places in the docs where we want that, instead of updating them all manually it should be a variable substitution.</p>
</blockquote>
<p>That is already the case. The <code>branchname</code> variable is filled in automatically (see <code>_includes/_branchname.liquid</code>) based on the <code>site.current_version</code> parameter which is set by the build scripts.</p>
<blockquote><blockquote>
<p>Liquid error: No such template ‘ssl_config’</p>
</blockquote>
<p>Uh oh?</p>
</blockquote>
<p>Ah, sorry fixed.</p>
<blockquote><blockquote>
<p>scp -r provision.sh local* tests user@host:</p>
</blockquote>
<p>So, we don't state explicitly that you're should be doing the git clone and config file editing on your local workstation. People who don't use Linux locally are more likely to start on the VM itself. In which case the <code>scp</code> step is unnecessary and potentially confusing.</p>
</blockquote>
<p>Yes, though, even on Windows you can do <code>git clone</code> now via WSL2. The text above that line did say "copy the files (if needed)". Anyway; I've added some more wording to address this. It's best practice to do the checkout on your workstation, and then copy things over before executing the provision script there. That way it is safe to blow away the target host and start over as needed.</p>
<blockquote>
<p>The key thing that needs to be stated clearly is that provision.sh is supposed to run on the host where everything will be installed.</p>
</blockquote>
<p>Yes, I've modified the text.</p>
<blockquote>
<a name="arvadosinstallsalt-multi-hosthtml"></a>
<h2 >arvados/install/salt-multi-host.html<a href="#arvadosinstallsalt-multi-hosthtml" class="wiki-anchor">¶</a></h2>
<blockquote>
<p>Copy the configuration files</p>
</blockquote>
<p>Maybe say "Copy and edit" or "Copy and customize" since they have to be updated.</p>
</blockquote>
<p>Done.</p>
<blockquote>
<p>If you try to run provision.sh without editing the file, will it give an error saying you didn't update stuff, or just crash?</p>
</blockquote>
<p>It tells you you need to modify parameters:</p>
<pre>
The config file /personal/projects/arvados/arvados/tools/salt-install/local.params has some parameters that need to be modified.
Please, fix them and re-run the provision script.
Error occurred (1) while running ./provision.sh at line 1 : exit 1
</pre>
<blockquote>
<p>Could the provision script have a subcommand for "copy self to target host, then run" ?</p>
</blockquote>
<p>I suppose it could.</p>
<blockquote>
<p>Could the provision script have a subcommand to distribute a new arvados config.yml to all the hosts?</p>
</blockquote>
<p>That's harder because it does not know about all your hosts. Every invocation of the script only knows about installing whatever you ask it to on whatever node you ask it to. Also; I think we <strong>do not</strong> want this; remember we advise people to stop running the installer once their cluster is up, cf. the last line on the single and multi page:</p>
<pre>
Re-running the Salt-based installer is not recommended for maintaining and upgrading Arvados, please see Maintenance and upgrading for more information.
</pre>
<p>Anyway; I do have another script to distribute config.yml, once the set of hosts is known. I could put that somewhere (where?).</p>
<p>Ready for another look at <a class="changeset" title="More Salt installer doc refactoring. refs #19175 Arvados-DCO-1.1-Signed-off-by: Ward Vandewege ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/8e2d796199b51ea5148c77be1a763fee96947a88">8e2d796199b51ea5148c77be1a763fee96947a88</a> on branch 19175-doc-refactor-multi-host-installation</p> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041452022-06-03T20:44:25ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>LGTM</p> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1041482022-06-04T00:50:52ZWard Vandewegeward@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset arvados-private:commit:arvados|7083622b12ec5f432fbe095c2230bde41bd84ca5.</p> Arvados - Feature #19175: [Documentation] refactor the multihost installer pagehttps://dev.arvados.org/issues/19175?journal_id=1090452022-12-13T15:32:11ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>47</i></li></ul>