https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-05-04T19:25:31ZArvadosArvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=923742021-05-04T19:25:31ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed parent" href="/issues/17512">Idea #17512</a>: Release Arvados 2.2</i> added</li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=924062021-05-05T15:54:34ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" href="/issues/17601">Bug #17601</a>: [deployment][arvados-formula] webshell needs a cron for login-sync</i> added</li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=924092021-05-05T15:56:43ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=924822021-05-10T15:46:37ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Related to shell nodes, it would be extremely useful if <code>sudo</code> could be set up to support the same PAM config as webshell, so that users can use valid Arvados tokens to authenticate to <code>sudo</code>.</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=925712021-05-12T15:52:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-12 sprint</i> to <i>2021-05-26 sprint</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=926912021-05-13T15:28:37ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>38</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=927452021-05-14T22:23:30ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Added a new state to manage <strong>Arvados resources</strong> through the formula.</p>
<p>These resources can be configured through the pillar, under <code>arvados.cluster.resources.virtual_machines</code></p>
<p>Initially, added an <code>arvados.controller.resources.virtual_machines</code> state file, to manage virtual machines and their scoped token as described in the documentation.</p>
<p>More resources can be added in the future easily.</p>
<p>The formula creates an entry for the virtual machine if it does not exists, checks for existing <code>scoped tokens</code> for it and adds one if none is found.</p>
<p>As both <code>virtual machines</code> and <code>scoped tokens</code> for them can be created by other means and Arvados does not check for their uniqueness, the formula uses the first existing entry for both resources if more than one are found.</p>
<p>I added checks to avoid creating these resources if they're found for a given machine.</p>
<p>The <strong>nginx</strong> configuration examples for these resources is still missing.</p>
<p>commit 1c81e52@arvados-formula, branch 17607-register-shell-nodes</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=927722021-05-17T18:30:39ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Added nginx examples to add multiple webshells (commit 5bf76f7@arvados-formula, 17607-register-shell-nodes)</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=927822021-05-17T22:14:15ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Added an example of some crude orchestration to get a shell node's <code>scoped token</code> and <code>vm uuid</code> and add them in a cron for <code>arvados-login-sync</code>.</p>
<p>As these tasks require <em>orchestration</em> (api server must be running before these queries being possible), I added them as examples, as they're useless on a single node configuration (which is the scope of a salt formula)</p>
<p>commit 4040230@arvados-formula, branch 17607-register-shell-nodes.</p>
<p>Need testing</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=927852021-05-17T22:19:53ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=928142021-05-18T18:52:57ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Some comments:</p>
<ul>
<li>File <code>arvados/controller/resources/virtual_machines.sls</code>
<ul>
<li>Line 45: Should '<code>fixme</code>' be on the regex? AFAICT this grep would always fail, or am I missing something? I'm seeing <code>fixme</code> being used on tests, but this <code>grep</code> command also would get executed on minions, correct? Shouldn't that regex use something like <code>{a-z0-9}[5]</code> instead?</li>
</ul>
</li>
<li>File <code>arvados/controller/service/running.sls</code>
<ul>
<li>Line 34-36: Is the <code>unless</code> clause necessary provided that it's using the same command execution check that que command itself?</li>
</ul>
</li>
<li>File <code>test/salt/states/examples/arvados/shell/cron/add-login-sync.sls</code>
<ul>
<li>Lines 58, 71, 81: Some refer to a file called <code>/tmp/sc-tk<vm></code> and other to a file called <code>/tmp/sctk-<vm></code> Are those references of the same file? If so, do you think it would be better to set the file name/path into a variable and use that?</li>
<li>Related to the above: Do you think it is feasible to randomize the use of temp files to avoid har dot debug issues?</li>
</ul></li>
</ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=928192021-05-18T19:43:04ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Some comments:</p>
<ul>
<li>File <code>arvados/controller/resources/virtual_machines.sls</code>
<ul>
<li>Line 45: Should '<code>fixme</code>' be on the regex? AFAICT this grep would always fail, or am I missing something? I'm seeing <code>fixme</code> being used on tests, but this <code>grep</code> command also would get executed on minions, correct? Shouldn't that regex use something like <code>{a-z0-9}[5]</code> instead?</li>
</ul></li>
</ul>
</blockquote>
<p>You're right. Fixed with a regex.</p>
<blockquote>
<ul>
<li>File <code>arvados/controller/service/running.sls</code>
<ul>
<li>Line 34-36: Is the <code>unless</code> clause necessary provided that it's using the same command execution check that que command itself?</li>
</ul></li>
</ul>
</blockquote>
<p><code>unless</code> and <code>onlyif</code> are control parameters in saltstack, that prevent/ensure a state to run if the check is true.</p>
<p>In this case, the <code>unless</code> provide a 'cosmetic' functionality: If the <code>unless</code> check passes, Salt won't 'run the state' and it won't be shown in the run report. Otherwise, it will be reported every time the state is run, which might be confusing.</p>
<blockquote>
<ul>
<li>File <code>test/salt/states/examples/arvados/shell/cron/add-login-sync.sls</code>
<ul>
<li>Lines 58, 71, 81: Some refer to a file called <code>/tmp/sc-tk<vm></code> and other to a file called <code>/tmp/sctk-<vm></code> Are those references of the same file? If so, do you think it would be better to set the file name/path into a variable and use that?</li>
</ul></li>
</ul>
</blockquote>
<p>Right, fixed the name to be consistent, thanks.</p>
<blockquote>
<ul>
<li>Related to the above: Do you think it is feasible to randomize the use of temp files to avoid har dot debug issues?</li>
</ul>
</blockquote>
<p>I don't wanted it to be a random name. In fact, I want it to be the same name: If the file is present, I don't want the state to run the command again. I we randomize it, every time the check will be run. And as we have to loop over all the <code>scoped tokens</code>, it's a costly run.</p>
<p>Squashed and force-pushed commit:79275e49|arvados-formula, branch 17607-register-shell-nodes</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=928232021-05-18T20:12:34ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Just one additional comment:</p>
<ul>
<li>In file <code>test/salt/states/examples/arvados/shell/cron/add-login-sync.sls</code> line 71 there's one more <code>sctk-<vm></code> file reference that seems to need correction. Note the dash '-', that isn't present on the other instances.</li>
</ul>
<p>Apart from that, it LGTM.</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=928252021-05-18T20:18:02ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Fixed at commit:e6dace0|arvados-formula , branch 17607-register-shell-nodes</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=931072021-05-26T16:10:46ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-26 sprint</i> to <i>2021-06-09 sprint</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=932182021-05-28T19:27:20ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>In Progress</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=932212021-05-28T19:38:39ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Blocks</strong> deleted (<i><a class="issue tracker-1 status-3 priority-4 priority-default closed parent" href="/issues/17601">Bug #17601</a>: [deployment][arvados-formula] webshell needs a cron for login-sync</i>)</li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=932462021-05-28T22:51:14ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>LGTM</p> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=935382021-06-09T15:45:19ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-06-09 sprint</i> to <i>2021-06-23 sprint</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=938532021-06-23T15:34:16ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-06-23 sprint</i> to <i>2021-07-07 sprint</i></li></ul> Arvados - Feature #17607: [deployment][provision][arvados-formula] register shell nodes with Arvadoshttps://dev.arvados.org/issues/17607?journal_id=939082021-06-23T20:47:13ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Merged</p>