https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-06-09T15:45:15ZArvadosArvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=935372021-06-09T15:45:15ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-06-09 sprint</i> to <i>2021-06-23 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=937022021-06-17T19:02:00ZJavier Bértolijbertoli@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/93702/diff?detail_id=90338">diff</a>)</li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=938522021-06-23T15:34:11ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-06-23 sprint</i> to <i>2021-07-07 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=945222021-07-07T15:30:27ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-07-07 sprint</i> to <i>2021-07-21 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=954752021-07-21T15:11:45ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-07-21 sprint</i> to <i>2021-08-04 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=958532021-08-04T15:29:30ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-08-04 sprint</i> to <i>2021-08-18 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=963002021-08-18T15:02:37ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-08-18 sprint</i> to <i>2021-09-01 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=966612021-09-01T15:15:16ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-09-01 sprint</i> to <i>2021-09-15 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=970512021-09-15T15:02:59ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-09-15 sprint</i> to <i>2021-09-29 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=973832021-09-28T19:16:15ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>42</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=974492021-09-29T15:09:24ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-09-29 sprint</i> to <i>2021-10-13 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=976302021-10-08T10:19:22ZJavier Bértolijbertoli@curii.com
<ul><li><strong>File</strong> <a href="/attachments/2905">salida_cluster_test_insecure_true</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2905/salida_cluster_test_insecure_true">salida_cluster_test_insecure_true</a> added</li><li><strong>File</strong> <a href="/attachments/2904">salida_cluster_test_insecure_false</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2904/salida_cluster_test_insecure_false">salida_cluster_test_insecure_false</a> added</li></ul><p>I'm experiencing some errors when running the final <code>cluster_tests</code> (attached)</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=976312021-10-08T10:33:53ZJavier Bértolijbertoli@curii.com
<ul><li><strong>File</strong> <a href="/attachments/2906">arvados_config.yml</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2906/arvados_config.yml">arvados_config.yml</a> added</li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=976932021-10-13T13:15:25ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Found the root cause of the error described above (a malformed <code>crunch-dispatch-local-credentials</code> file, fixed in arvados-formula:commit:ba3827)</p>
<p>Refactored the provisioning script and configuration files:</p>
<ul>
<li>pillars: SSL certs and keys are included now on each pillar, and not as a snippet, so we can iterate over them in the provision script</li>
<li>split <code>keepweb</code> nginx's configuration in its two parts (<code>download</code> and <code>collections</code>) for the same reason.</li>
<li>added a new state (<code>custom_certs</code>) which copies the certs from a dir where the user uploads them.</li>
<li>refactored the <code>snakeoil_certs</code> in <code>/single_host/multiple_hostnames</code> to generate individual certs that can be "uploaded" by <code>custom_certs</code>, so we can test it.</li>
<li>added a <code>--development</code> parameter to the the <code>provision.sh</code> script, to include the <code>snakeoil</code> certs when testing.</li>
<li>added documentation.</li>
</ul>
<p><a class="changeset" title="17742: add documentation about custom certs usage Arvados-DCO-1.1-Signed-off-by: Javier Bértoli ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/15a2556dc2f45b393641a9ee012306dacb3edd5c">15a2556dc</a> (branch 17742-provide-custom-certs)</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=977492021-10-13T15:49:00ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-10-13 sprint</i> to <i>2021-10-27 sprint</i></li></ul> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=977932021-10-15T21:01:52ZWard Vandewegeward@curii.com
<ul></ul><p>Reviewing <a class="changeset" title="17742: add documentation about custom certs usage Arvados-DCO-1.1-Signed-off-by: Javier Bértoli ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f54cc984969657be50c093b917feb49a19d78c22">f54cc984969657be50c093b917feb49a19d78c22</a></p>
<p>In doc/install/salt-multi-host.html.textile.liquid</p>
<pre>
+The <i>multi_host</i> include LetsEncrypt salt code to automatically request and install the certificates for the public-facing hosts (API/controller, Workbench, Keepproxy/Keepweb) using AWS' Route53.
</pre>
<p>There seems to be a word missing before "include".</p>
<pre>
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):
</pre>
<p>Typo: "download".</p>
<p>doc/install/salt-single-host.html.textile.liquid</p>
<p>Does this method not include Let's Encrypt salt code? It doesn't seem mentioned in that file.</p>
<pre>
+The script expects cert/key files with these basenames (matching the role except for <i>keepweb</i>, which is split in both <i>downoad / collections</i>):
</pre>
<p>Typo: "download"</p>
<p>In tools/salt-install/config_examples/single_host/multiple_hostnames/pillars/arvados.sls</p>
<pre>
# required to test with arvados-snakeoil certs
- insecure: true
+ insecure: false
</pre>
<p>Since you're changing it, is that comment still relevant? If so what does "test" mean? Automated tests? A user trying out Arvados? Something else? Is this what you mean: "When using arvados-snakeoil certs set insecure: true" ?</p>
<p>In tools/salt-install/provision.sh</p>
<pre>
+ grep -q ${R} ${P_DIR}/extra_custom_certs.sls || echo " - ${R}" >> ${P_DIR}/extra_custom_certs.sls
+
+ # As the pillar differ whether we use LE or custom certs, we need to do a final edition on them
+ # Special case for keepweb
</pre>
<p>That empty line has trailing whitespace...</p>
<p>I haven't tried it yet.</p>
<p>Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=978042021-10-18T23:05:55ZJavier Bértolijbertoli@curii.com
<ul></ul><p>Ward Vandewege wrote:</p>
<blockquote>
<p>Reviewing <a class="changeset" title="17742: add documentation about custom certs usage Arvados-DCO-1.1-Signed-off-by: Javier Bértoli ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f54cc984969657be50c093b917feb49a19d78c22">f54cc984969657be50c093b917feb49a19d78c22</a></p>
</blockquote>
<p>Addressed your suggestions on <a class="changeset" title="17742: update script addressing review suggestions Arvados-DCO-1.1-Signed-off-by: Javier Bértoli..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/d68c3776fee61329f665ede740cca51946395d1c">d68c3776f</a> (branch 17742-provide-custom-certs)</p>
<blockquote>
<p>I haven't tried it yet.</p>
<p>Is the plan to test the custom cert functionality with the automated test-deploy jobs on Jenkins?</p>
</blockquote>
<p>Yes, the way I modified the code, the <em>snake oil certificates</em> are deployed using the custom-certs functionality:</p>
<p>The <a href="https://git.arvados.org/arvados.git/blob_plain/HEAD:/tools/salt-install/config_examples/single_host/multiple_hostnames/states/snakeoil_certs.sls" class="external">snakeoil_certs</a> state file now creates certificates that are deployed in the destination directory where the <a href="https://git.arvados.org/arvados.git/blob_plain/refs/heads/17742-provide-custom-certs:/tools/salt-install/config_examples/single_host/multiple_hostnames/states/custom_certs.sls" class="external">custom_certs</a> state file expect them to be (and where the docs specify they should be copied). From there, the latter will deploy them.</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=978602021-10-21T18:44:01ZWard Vandewegeward@curii.com
<ul></ul><p>Thanks, a few more things:</p>
<ul>
<li>Please fix the spelling of <code>Let's Encrypt</code>, you have "LetsEncrypt" in three places in the docs, and that is wrong.</li>
</ul>
<ul>
<li>The blurb about custom certificates in `doc/install/salt-multi-host.html.textile.liquid` and `doc/install/salt-single-host.html.textile.liquid` appears to be identical, please pull that out into an 'include', see the <code>_includes</code> directory for an example.</li>
</ul>
<p>With those changes, LGTM thanks.</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=978662021-10-21T19:02:13ZJavier Bértolijbertoli@curii.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '17742-provide-custom-certs' closes #17742 Arvados-DCO-1.1-Signed-off-by: Javier Bé..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9539317a22d8ea16f94b0e086507ab595d758216">arvados|9539317a22d8ea16f94b0e086507ab595d758216</a>.</p> Arvados - Feature #17742: [deployment][provision] allow to provide custom SSL certificateshttps://dev.arvados.org/issues/17742?journal_id=988952021-11-23T16:13:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> changed from <i>42</i> to <i>45</i></li></ul>